Bake a calibrated model into quantized inference modules.
convert is the final step of the static-PTQ pipeline (prepare →
calibrate → convert). It walks the calibrated model and swaps every
observed float submodule for its quantized counterpart: the learned float
weight is quantized once into int8 codes plus per-channel scale /
zero_point (dropping the float weight), and each activation observer's
recorded range is frozen into that layer's output qparams. What comes back
is a self-contained, eval-mode model with no observers left — ready to
run or serialize, with a checkpoint roughly 4x smaller than the float
original.
A submodule is convertible when calibration attached an
activation_post_process to it; a small always-convert set
(DeQuantStub, Embedding, EmbeddingBag) converts even without an
activation observer. Replacements are spliced in through _modules rather
than setattr so a Sequential's child order is preserved.
When lucid.backends.quantized.use_mlx() is true (the MLX quantized engine
is present), the Linear family is routed to the real weight-only
quantized_matmul GEMM — the genuine compute + memory win, at W8A16
numerics (weights int8/int4, activations kept float) — instead of the
dequantize-to-float reference path. On a fully-Metal model this is bandwidth-
bound and measures ~3.15x faster at the memory-bound decode shape
M = 1 with a weight payload ~3.55x smaller; layers whose
in_features is not divisible by a supported MLX group size fall back to
the dequant path automatically. Set backends.quantized.engine = "reference" to force the exact W8A8 dequant numerics on every layer.
Parameters
modelnn.Moduleprepare and then run through
calibration, so every quantizable submodule carries frozen activation
qparams. QAT models (from prepare_qat, fine-tuned) also convert
here — their nn.qat layers map to the same quantized inference layers.inplacebool= FalseTrue mutate and return model itself; if False (default)
convert a copy.deepcopy and leave the calibrated model intact.Returns
nn.ModuleThe converted, eval-mode model with int8 weights baked in and its
float submodules replaced by their quantized counterparts.
Notes
- Non-destructive by default (deep-copy); pass
inplace=Trueto convert the calibrated model in place and reclaim its float weights. - Only int8 codes + qparams + float bias enter the resulting
state_dict; the float weight never does (a whole checkpoint shrinks ~3.97xonresnet_18). - The reference (non-MLX) path wins on memory, not compute — its GEMM runs in float and so is roughly float-speed on the CPU stream. Convert with the MLX backend active for the genuine low-precision compute speed-up.
- Calling
converton a model that was never calibrated is a silent correctness bug: the output qparams are whatever the fresh observers hold (often identity), so run calibration data throughprepare's output first.
Examples
>>> import lucid, lucid.nn as nn
>>> import lucid.quantization as Q
>>> model = nn.Sequential(nn.Linear(128, 64))
>>> model.qconfig = Q.get_default_qconfig()
>>> prepared = Q.prepare(model)
>>> for _ in range(8): # calibrate before converting
... _ = prepared(lucid.randn(32, 128))
>>> qmodel = Q.convert(prepared)
>>> type(qmodel[0]).__name__
'Linear'
The common mistake — converting straight after prepare with **no**
calibration — silently produces a model with meaningless activation qparams:
>>> prepared = Q.prepare(nn.Sequential(nn.Linear(128, 64)))
>>> qmodel = Q.convert(prepared) # no data ran through -> garbage output scaleSee Also
- lucid.quantization.prepare—Insert observers ahead of calibration.
- lucid.quantization.convert_fx—The graph-mode counterpart.
- lucid.quantization.quantize_dynamic—Calibration-free alternative.
- lucid.nn.quantized.Linear—The quantized layer this installs for Linear.