Embedding

Quantized embedding lookup table — per-row int8, dequantize-on-lookup.

The inference-time replacement that lucid.quantization.convert installs for a calibrated float lucid.nn.Embedding. Embedding tables dominate the parameter budget of large-vocabulary language models — a (50000, 768) table is ~150 MB in float32 — so storing the table as int8 is the single largest memory win quantization offers on the text side, and this layer is where it lands.

Representation. The learned table (num_embeddings, embedding_dim) is quantized once, at from_float time, into an int8 code tensor plus a per-row (per-token) scale / zero_point; the float table is then dropped. Because each vocabulary row gets its own scale, tokens with very different embedding magnitudes are all tracked tightly — far more accurately than a single per-tensor scale. Each forward dequantizes the table back to float and runs the ordinary lucid.nn.functional.embedding lookup, so the layer needs no int8 gather kernel and runs unchanged on any device.

Encoding happens once; decode + gather run on every forward. For row $i$, column $j$:

where $s_i, z_i$ are the row-$i$ scale / zero-point (symmetric qint8, so $z_i = 0$) and $x_k$ the $k$-th input index. The lookup gathers dequantized rows; only rows actually indexed are touched.