Quantisation shrinks an AI model by rounding off the values that define its behaviour. A neural network is just a giant list of weights, usually stored at high precision. Quantisation compresses these into smaller formats. This drastically cuts the memory needed to run the model, allowing massive systems to fit on standard hardware.
The cost is a slight loss in precision. For basic chat, this rounding is mostly invisible. For complex reasoning, the model becomes noticeably less capable. You trade a little intelligence for massive hardware savings.
The short version- Quantisation shrinks a model by rounding the weights that define its behaviour.
- It cuts memory use drastically, allowing massive models to run on standard hardware.
- Moderate rounding is almost lossless for general text tasks.
- Aggressive rounding degrades complex reasoning, exact arithmetic and rare vocabulary.
- Most downloaded models are compressed after training rather than trained to tolerate the rounding.
What precision means for one weight
A neural network is a giant list of weights. In their original form, these are stored as floating-point formats that keep several significant digits. Quantisation is just clever rounding. It compresses these precise weights into smaller integers.
When you drop to these smaller integers, you are no longer storing decimals. You are snapping weights to one of a few distinct levels. The weight survives the conversion, but its exactness does not. We can see this by tracking a single weight as it survives each format.
a single weight shown at 32-bit, 16-bit, 8-bit and 4-bit precisionDuring training, the model tweaks this connection strength by tiny fractions to learn subtle differences. To capture these microscopic adjustments, the system needs high precision. Floating-point formats provide this by using a sliding decimal point. This allows the computer to store extremely small fractions accurately.
When we quantise, we throw away this sliding scale. We force the weight to snap to the nearest available notch on a fixed grid. In a neural network, snapping to a grid means the connection strength is no longer exactly what the training process decided it should be. The model becomes slightly clumsier.
Smaller weights also answer faster
The bottleneck of running artificial intelligence is rarely the processor itself. The processor is often fast enough to perform the necessary calculations instantly. The true bottleneck is the time it takes to move data from the memory chips to the processor. Every single time the model generates a single word, it must read its entire list of weights.
Moving that much data back and forth requires immense memory bandwidth. Imagine a brilliant chef working in a kitchen with a tiny serving hatch. The chef can cook incredibly fast, but they spend most of their time waiting for ingredients to be passed through the hatch. In a server, the processor is the chef, and the memory bandwidth is the hatch.
Language models are uniquely starved for these ingredients. They do very little actual math per weight, but they must touch every single weight to generate a single piece of text. By shrinking each weight, we effectively widen the serving hatch. The processor spends less time waiting and more time generating.
The overhead of unpacking the compressed weights is tiny compared to the time saved by moving less data. This physical reality is the fundamental reason why local inference is possible today. Without compression, the memory bandwidth of consumer hardware would restrict text generation to an unusable crawl. The entire ecosystem relies on this trade-off.
Bandwidth over computeA processor can only generate text as fast as the memory can feed it weights, making file size the ultimate speed limit.
The memory arithmetic
The memory required to run a model is a hard floor. You multiply the bytes per parameter by the total parameter count. This strict arithmetic is why a model that crashes a server at full precision can run on a laptop when heavily compressed. If the total size of the compressed weights exceeds your available memory, the model simply will not load.
memory required to load an 8-billion-parameter model at different quantisation levelsWhen you download a quantised model, it carries a small overhead. This overhead accounts for the scaling factors that help the rounded weights make sense. Even with this addition, the savings are massive. Arithmetic checked 5 September 2026.
4.5 GBThe memory required to fit an 8B model at INT4 precision, down from 32 GB.
Why outliers dictate model intelligence
In any large network, most weights hover near zero. They are unremarkable, representing weak connections between concepts. These small weights are easily rounded without consequence. However, a tiny fraction of weights grow unusually large during the training process.
These outliers act as load-bearing pillars for the model's logic. They often control rigid, rule-based behaviour, such as mathematics, formatting, or strict logical deductions. If you apply a naive grid to the entire model, these massive outliers get crushed. They are forced into the highest available notch, losing their distinct scale relative to the rest of the network.
When the model tries to rely on these crushed pillars, the logic collapses. The network becomes confidently incorrect, generating perfectly fluent text that contains fatal logical flaws. Modern compression techniques solve this by grouping weights into small blocks and calculating a unique scale for each block. This block-based approach allows the grid to stretch and accommodate outliers without ruining the precision of the normal weights nearby.
Modern compression techniques solve this by grouping weights into small blocks and calculating a unique scale for each block. — to protect the outliers that control logic.
Where the rounding hurts
Quantisation error is not spread evenly across the model's capabilities. Moderate rounding is close to lossless for most models. Aggressive rounding remains perfectly fine for general chat, summarising, and extracting text. Below that threshold, the quality falls quickly and noticeably.
how capability drops as precision falls, showing reasoning failing before basic chatThe losses show up first in rigid tasks. Arithmetic, long multi-step reasoning, rare words, and code generation degrade before basic conversation does. Language is fluid and robust, meaning the redundancy in language protects it from rounding errors. This is why basic chat survives aggressive compression.
Logic, however, is brittle. A mathematical calculation or a block of code does not have redundancy. If a single variable name is hallucinated, the entire script crashes. If a single step in a reasoning chain is flawed, the final conclusion is wrong.
How models are compressed
There are two ways to apply this rounding. Post-training quantisation takes a finished, fully trained model and rounds its weights down. It uses a small calibration dataset to measure which weights matter most before deciding how to round them. This takes minutes or hours, making it accessible to the community.
Quantisation-aware training takes a different path. It bakes the rounding directly into the training process, forcing the model to learn to tolerate the reduced precision from the start. This yields better quality, but it requires the massive compute of a training run. Almost all the quantised files people download are produced by post-training quantisation.
When a massive technology company releases a new model, they typically release the uncompressed version. They do not release the versions that have been trained to tolerate compression. It falls to the open-source community to download the massive file, run the calibration process, and publish the compressed variants. This community-driven pipeline is why you see dozens of different compressed variants appear within hours of a major release.
When you browse downloadable models, you will hit a wall of acronyms. GGUF is the container used by llama.cpp and the tools built on it, designed for ordinary processors and mixed CPU-and-GPU setups. It packages the weights, the tokeniser and the configuration into a single file. Names like Q4_K_M say how the file was compressed: the base precision, mixed precision for sensitive layers, and the size-to-quality balance.
a guide to GGUF, AWQ and GPTQ and the hardware they targetIf you run on dedicated graphics cards, you will meet AWQ and GPTQ instead: weight-only schemes built for the way a GPU reads memory. Dedicated graphics cards have their own unique architecture and read memory in wide, parallel blocks. If you use a compression scheme designed for a standard processor, the graphics card will stall while trying to unpack the weights. The alternative formats are engineered specifically for this parallel reading.
They arrange the compressed weights and their scaling factors in a way that allows the graphics card to unpack them instantly. They actively protect the model by keeping a small percentage of the most important weights at higher precision. This keeps the processor fed and the text generating at high speed. We verified these formats against their respective project documentation, noting the exact capabilities at the time of our research.
Making the business trade-off
For a business, the practical question is whether the compressed model still answers your customers correctly. You must test the compressed model on your own real questions before trusting it. We route simple questions to a small fast model and hard questions to a larger one. The queries a customer asks a business are exactly the ones a heavily quantised small model answers well.
Measured, not estimated: We route cheap turns to a small fast model and hard turns to a larger one, verified in production.We red-teamed our own public assistant and reproduced several faults. The failures that matter to a customer come from rules, retrieval, and escalation, not from the last few bits of precision. A customer asking for a refund policy does not require deep multi-step reasoning. They need a clear, retrieved fact, which a compressed model delivers flawlessly.
131Conversations red-teamed in our own public assistant1
33Faults reproduced and investigated2
27Faults fixed3
14Rules added4
1Code defect corrected5
1. iHayz production log · 2. iHayz production log · 3. iHayz production log · 4. iHayz production log · 5. iHayz production log
If managing these formats sounds exhausting, a hosted service hides all of this from you. For a small team, paying a fraction of a cent per chat is usually the right trade to avoid managing hardware. The most efficient systems do not rely on a single model. This hybrid approach gives you the speed and cost savings of compression where it matters, and high-precision logic only when it is strictly necessary.
Knowledge Assistants — Website Assistant from $699 build + $79/mo, including 500 conversations a month
SSathish Balakrishnan · Founder, iHayzBuilds and operates AI agents for small and mid-size businesses — and runs every one of them on his own accounts first. Everything in writing, no calls.
Questions people ask
5 answers, written from the article01Does aggressive quantisation make the model stupid?
Not for basic tasks. It degrades complex reasoning and coding, but handles general chat and summarisation very well.
02What does Q4_K_M mean?
Q4 is the 4-bit base precision, K is the K-quant scheme that keeps sensitive layers at higher precision, and M is the medium balance of size and quality.
03Should I use GGUF, AWQ or GPTQ?
GGUF on a standard processor, a laptop or a mixed CPU-and-GPU machine. AWQ or GPTQ when the whole model sits on dedicated graphics cards.
04Why not just use moderate compression for everything?
Moderate compression halves your memory requirement, but aggressive compression halves it again. If you are constrained by hardware, aggressive compression is often the only way a model will fit.
05Are hosted models quantised?
Yes, behind the scenes, major providers heavily optimise and quantise their models to serve them profitably, though they do not publish the exact specifications.