LinearReLU

Quantization-aware fused Linear + ReLU — trainable, fake-quant every forward.

A Linear whose forward applies ReLU before the output fake-quant, so the activation observer sees the true post-ReLU (non-negative) range and calibrates a grid that spends all of its codes on values the layer can actually emit. Fusing the two ops also lets a single activation observer stand in for both, which is exactly the shape lucid.quantization.convert needs to collapse them into one kernel.

Like the base Linear, the weight and the (post-ReLU) output are fake-quantized every forward under a straight-through estimator (STE): rounding happens in the forward pass while gradients pass straight through to the float weight, so it stays fully trainable and learns to compensate for the eventual int8 rounding. Built from a fused float lucid.nn.intrinsic.LinearReLU by lucid.quantization.prepare_qat; lucid.quantization.convert folds it into a single quantized lucid.nn.quantized.LinearReLU.

On each forward the weight is fake-quantized, F.linear runs, ReLU clips, and the result is fake-quantized onto the observed post-ReLU grid:

where $s, z$ are the observer's scale / zero-point and $q_{\min}, q_{\max}$ the grid bounds. Because the activation observer sits outside the ReLU, the calibrated $(s, z)$ describe the non-negative output.