| Quantization | Size relative to FP16 | Quality | Use case | |--------------|----------------------|---------|-----------| | q4_0 / q4_1 | ~25% (small) | lower | fast CPU | | | ~30% (medium) | good | balanced | | q8_0 | ~50% (large) | better | higher accuracy |
This model is often chosen as the "sweet spot" for users who need a balance between professional accuracy and processing speed.
Using fewer threads than cores or a non-optimized build. Fix: