Choose LLM Quantization for Your Task

Choose a quantization by testing whether it preserves the outcomes your task requires within your memory budget. A smaller file can make a model practical, but a format label alone does not tell you how accurately it will extract dates, write code or follow your output rules.

Compare the same underlying model

Hold the model family, version, prompt format and runtime constant. Comparing a small quantization of one model with a larger quantization of another mixes compression effects with differences in training and architecture.

The upstream llama.cpp quantization discussion contains task-dependent user observations. Treat those reports as reasons to test your workload, not as a universal accuracy table.

Keep the exact file identifiers. A filename containing a familiar quantization label is not enough to establish that two files came from the same base weights or conversion pipeline.

Build a compact acceptance set

Select examples from the work you actually do. For a support assistant, include a straightforward answer, a misleading document, a missing fact and a required refusal to guess. For coding, include compilation, behavior and an edge case.

Write the acceptable outcome before generating answers. Otherwise a fluent result can change your standard after the fact.

Suggested worksheet:

Case	Required outcome	Larger candidate	Smaller candidate	Correction needed
Known fact	Exact supported value	Record	Record	Record
Missing fact	Explicitly unknown	Record	Record	Record
Structured result	Valid and accurate fields	Record	Record	Record
Hard edge case	Stated invariant holds	Record	Record	Record

This is a proposed test design, not a claim about any model's score.

Include operational costs

Measure memory, loading time and completion time with the same settings. Also record human correction time. Faster output that repeatedly needs repair may be a poor trade.

If the larger candidate cannot fit on your hardware, do not present a comparison as though both ran under equal conditions. State the limitation and compare the smaller candidate with your actual acceptance threshold.

Repeat failures to understand variability, but preserve the original failure. Selecting only the best attempt makes a weak configuration look stronger than it is.

Make a bounded decision

An acceptable conclusion is: “This configuration passed these document-extraction cases at this context on this machine.” Avoid “Q4 is always enough” or “Q8 is lossless.”

Use prompt regression testing to preserve the cases for later changes. The 16GB RAM guide explains the hardware constraints, while this test decides whether the smaller configuration remains useful.

Related troubleshooting

• Ollama Out of Memory in Longer Chats
• Valid AI JSON Can Still Be Wrong

This guide draws on the linked documentation. Examples are illustrative unless explicitly identified as measured results.