This page may contain stale information. Last updated: 2026-04-21
Definition
Quantization is a compression technique that reduces the numerical precision of neural network weights and activations. Converting from 32-bit floats to lower precision (8-bit, 4-bit, 3-bit) reduces model size and memory bandwidth requirements.
Key Tradeoff
- Standard precision (FP32): Full accuracy, high memory/compute cost
- Quantized (INT8, INT4, INT3): Reduced accuracy risk, but 4-8x smaller models and faster inference
Quantization Approaches
- Post-Training Quantization (PTQ): Apply after training (no retraining required)
- Quantization-Aware Training (QAT): Include quantization during training
- Dynamic Quantization: Different precision for different layers/weights
Challenge: Maintaining Accuracy
Historical problem: Lower precision = accuracy degradation
- INT8 often acceptable (1-2% accuracy loss)
- INT4 requires careful calibration
- INT3 previously considered impractical
Breakthrough: PolarQuant
google’s turboquant introduces novel approach (polarquant) enabling 3-bit quantization without accuracy loss. Specifically targets kv-cache values, where redundancy is high.
Production Applications
- Model inference (cost reduction)
- Mobile/edge deployment (size reduction)
- Long-context LLMs (memory efficiency)
- Vector search indices (faster retrieval)