MovingAverageMinMaxObserver
MinMaxObserverMovingAverageMinMaxObserver(qscheme: QScheme = per_tensor_affine, qdtype: QDtype = quint8, averaging_constant: float = 0.01, eps: float = 1e-08)Per-tensor observer with an exponential moving average of min / max.
An outlier-tolerant refinement of MinMaxObserver. Instead of taking a hard
cumulative min/max, it tracks an exponential moving average of each batch's range,
so a single anomalous batch nudges the running range only slightly and is soon
forgotten as later batches pull it back toward the bulk of the data. The result is a
tighter, more representative grid than a plain min/max on noisy activation streams —
at the cost of one hyperparameter, the averaging constant.
When to pick it. Prefer this over MinMaxObserver whenever calibration
data may contain occasional spikes (most real activation streams do) and you want a
range that reflects the typical extreme rather than the absolute worst case. It is
the activation observer wrapped by FakeQuantize in
lucid.quantization.get_default_qat_qconfig, where smoothing also stabilises
training. For heavy-tailed distributions where you would rather clip the tail than
average it, HistogramObserver is stronger; for weights, use
MovingAveragePerChannelMinMaxObserver.
The first batch seeds the range directly (the ±inf seed cannot be averaged);
every later batch applies the EMA update with averaging constant :
Smaller means heavier smoothing (slower to react, more outlier-robust);
degenerates to using only the latest batch. The reduction to
(scale, zero_point) is the same affine / symmetric map inherited from
MinMaxObserver.
Parameters
[0, 255] grid).averaging_constantfloat= 0.01epsfloat= 1e-8scale.Attributes
Notes
- First-batch seeding.
forwarddetects the+infseed and copies the first batch's range in verbatim; the EMA recurrence applies only from the second batch on. Without this the average would be corrupted by the infinite seed. - Not cumulative. Unlike
MinMaxObserver, the range can shrink between batches — this is exactly why a lone outlier does not permanently widen the grid. - Buffer re-registration. The EMA update is written back via
register_buffer(same name), per theObserverBasecontract. - Symmetric vs affine behaves as in the base class;
averaging_constantonly affects the running range, not the range → qparams reduction. - Default activation observer of
lucid.quantization.get_default_qat_qconfig(wrapped inFakeQuantize).
Examples
>>> import lucid
>>> import lucid.quantization as Q
>>> obs = Q.MovingAverageMinMaxObserver(averaging_constant=0.1)
>>> for _ in range(8):
... _ = obs(lucid.randn(16, 128))
>>> scale, zero_point = obs.calculate_qparams()
>>> bool(scale.item() > 0)
True
A lone outlier batch barely moves the smoothed range (contrast the hard-min/max
observer, which would absorb the full spike):
>>> obs = Q.MovingAverageMinMaxObserver(averaging_constant=0.01)
>>> _ = obs(lucid.zeros(1, 4)) # seed range ~ [0, 0]
>>> _ = obs(lucid.full((1, 4), 1000.0)) # one huge batch
>>> bool(obs.max_val.item() < 20.0) # EMA absorbed only ~1% of the spike
TrueSee Also
- lucid.quantization.MinMaxObserver—Hard cumulative min/max (no smoothing).
- lucid.quantization.MovingAveragePerChannelMinMaxObserver—Per-channel EMA analogue.
- lucid.quantization.HistogramObserver—Tail-clipping alternative for outliers.
- lucid.quantization.get_default_qat_qconfig—Uses this observer for activations.
- lucid.quantization.FakeQuantize—Wraps it for quantization-aware training.