LinearReLU
LinearLinearReLU(in_features: int, out_features: int, bias: bool = True)Quantized linear layer with a fused ReLU — int8 weight, float compute.
A fused (intrinsic) block: the quantized counterpart of the float
nn.intrinsic.LinearReLU, installed by lucid.quantization.convert
/ from_float. It behaves exactly like
lucid.nn.quantized.Linear — int8 per-output-channel weight, float
matmul, output fake-quantized to the calibrated grid — but folds a ReLU into
the layer so the linear and its activation share a single quantized output.
Why fuse. During calibration the fused module's activation observer sat
after the ReLU, so it recorded the post-ReLU (non-negative) range. To
reproduce those numerics at inference the ReLU must be applied before the
output fake-quant — exactly what the overridden _activation hook does.
Fusing also requantizes the whole linear → relu pair once, against the
tighter non-negative range, which uses the int8 grid more efficiently than
quantizing the raw linear output and clamping afterwards.
Encoding (int8 weight) happens once at from_float; decode + matmul +
fused ReLU + requantize run on every forward:
where are the per-output-channel weight scale / zero-point,
the float bias, and the outer fake-quant snaps the post-ReLU result
to the scalar calibrated output (scale, zero_point).
Parameters
in_featuresintin dimension of the weight.out_featuresintout dimension, quantized per-channel.biasbool= TrueTrue.Attributes
weight_int8Tensorint8 weight codes, shape (out_features, in_features).weight_scaleTensor(out_features,).weight_zero_pointTensor(out_features,).scaleTensorzero_pointTensorbiasTensor or None(out_features,), or None.Notes
- Produced by
convert/from_float, not constructed directly; the constructor is inherited unchanged fromlucid.nn.quantized.Linearand only the_activationhook differs. from_floataccepts either a float fused module (annn.Sequentialwhose[0]is the weighted layer) or a single-layer QAT fused module; the shared wiring copies the fused qconfig / observer onto the weighted child before quantizing.- The output range is non-negative because the ReLU precedes the fake-quant — calibrate the fused module (not the bare linear) so the observed grid matches what runs at inference.
Examples
>>> import lucid, lucid.nn as nn
>>> qlr = nn.quantized.LinearReLU(64, 32) # normally built via convert()
>>> qlr(lucid.randn(4, 64)).shape
(4, 32)
A directly constructed instance carries zeroed int8 weights and identity
qparams — like Linear, it returns garbage until from_float /
load_state_dict populates the buffers:
>>> bool((qlr.weight_int8 == 0).all().item())
TrueSee Also
- lucid.nn.quantized.Linear—The un-fused quantized linear it extends.
- lucid.nn.quantized.ConvReLU2d—The convolutional fused analogue.
- lucid.quantization.convert—Installs this layer from a calibrated model.
Used by 1
Class methods
1from_float(mod: nn.Module)Build a QAT Linear from a float one (shares the trained weight).