What is quantization in AI models?

Quantization is a technique that makes AI models smaller and faster by storing their internal numbers with less precision. It is the main reason large language models can run on ordinary laptops and even phones.

To understand it, remember that an AI model is essentially billions of numbers, called weights. Normally, each weight is stored with high precision, using 16 or 32 bits of memory. Quantization rounds these numbers to lower precision, such as 8 bits or 4 bits.

Here is a simple analogy. Imagine storing a price as 1,234.5678 rupees versus 1,235 rupees. The rounded version takes less space and is good enough for most purposes. Quantization does something similar across billions of numbers.

The savings are large. A 7-billion-parameter model at 16-bit precision needs roughly 14 GB of memory. At 4-bit precision, it needs around 4 to 5 GB. Suddenly, a model that needed a powerful GPU can run on a normal laptop.

When you download models for tools like Ollama or LM Studio, you will see labels such as:

GGUF: a popular file format for quantized models used by llama.cpp and tools built on it.
Q8: 8-bit quantization, very close to original quality, but larger.
Q5 and Q6: good quality with moderate size.
Q4: a popular balance of size and quality for most users.
Q3 and Q2: very small, but with noticeable quality loss.

You may also see names of other quantization methods, such as GPTQ, AWQ or MLX formats for Apple Silicon. They are different techniques with the same goal.

What is the trade-off? Lower precision means some loss of quality. The model may be slightly less accurate, make more mistakes in complex reasoning or follow instructions less reliably. With good modern methods, 4-bit and 5-bit models often perform close to the original for everyday tasks. Below 4 bits, the drop usually becomes more noticeable.

A practical rule: a larger model with moderate quantization often performs better than a smaller model at full precision using the same memory. For example, a quantized 14-billion-parameter model may beat an unquantized 7-billion one.

Quantization also speeds up responses and reduces costs on servers, which is why it is used in production systems, not just on laptops.

If you are new to local AI, start with Q4 or Q5 versions, which give the best balance for most computers.