Post-Training Quantization
Imagine taking a model that works well on a workstation and packing it into a much smaller suitcase without retraining it from scratch. Post-Training Quantization (PTQ) does this by converting a finished model’s high-precision numbers into compact, hardware-friendly ones after its original training is complete.
What changes inside the model
Most models are trained with 32-bit floating-point values (FP32) for weights and intermediate calculations. PTQ commonly converts these values to 8-bit integers (INT8). Instead of storing every value with a large floating-point representation, the runtime stores a small integer plus scale and zero-point information that maps it back to an approximate real-world value. This cuts model storage roughly fourfold and lets compatible processors use fast, energy-efficient integer math.
Calibration protects useful accuracy
For weights, conversion can be calculated directly from their values. Activations—the temporary outputs produced while the model runs—need more care. In static PTQ, developers pass a small, representative calibration dataset through the model to measure realistic activation ranges and choose good quantization scales. Dynamic quantization instead determines activation ranges while running, which is simpler but can add overhead. Per-channel quantization gives each output channel of a layer its own scale, preserving accuracy particularly well for convolutions.
Why it matters on devices
PTQ is a practical deployment step for a smart camera, phone, or Cortex-M-based sensor because it needs no expensive retraining cycle. Tools such as TensorFlow Lite, ONNX Runtime, and Core ML can export or execute quantized models, while mobile NPUs and microcontroller accelerators are designed around INT8 operations. A wake-word model can therefore remain active within a battery budget, or an offline camera can detect objects with less memory and heat. The trade-off is accuracy: a poorly chosen calibration set, or a model with unusually sensitive layers, can cause noticeable prediction errors. When that loss is unacceptable, quantization-aware training teaches the model to tolerate low-precision arithmetic during retraining.
Post-Training Quantization (PTQ) converts an already trained model from floating-point values to lower-precision formats such as INT8, using calibration data to set quantization ranges without retraining. It reduces model size, memory bandwidth, and inference latency while preserving acceptable accuracy. For edge deployment, PTQ enables existing models to run efficiently on integer-accelerated phones, embedded processors, and microcontrollers with minimal conversion effort.
Imagine packing for a trip with a small suitcase. Instead of bringing every bulky item, you replace some with lighter versions that still do the job. Post-training quantization does something similar for an AI model after it has already learned.
It converts the model’s stored numbers into simpler, smaller ones. This can make the model take up less storage, use less battery, and often run faster on a phone, camera, or sensor. The goal is to preserve useful answers while accepting, at most, a small drop in accuracy. It helps bring capable AI onto everyday devices without needing a constant internet connection.