SinusoidalEmbedding
ModuleSinusoidalEmbedding(num_positions: int, embedding_dim: int, base: float = 10000.0)Fixed 1-D sinusoidal positional encoding (Vaswani et al., 2017).
Carries no learnable parameters — the encoding table is precomputed once
in __init__ and stored as a non-persistent buffer. forward()
returns the full (num_positions, embedding_dim) table; callers
typically slice it with pe()[:seq_len] and broadcast against their
batched hidden states.
Parameters
num_positionsintembedding_dimintbasefloat= 10000.010000
per Vaswani 2017; some long-context models use larger. Default
10000.0.Notes
The encoding table is materialised once in __init__ and registered
as a non-persistent buffer; it is therefore not learnable and is not
saved in state_dict (but moves with .to(device=...)). Prefer
the module form when the same maximum sequence length is reused across
many forward passes; use the functional
lucid.nn.functional.sinusoidal_embedding form for one-shot or
variable-length encodings where caching is wasteful. See that function
for the underlying sin/cos formula.
Examples
>>> import lucid.nn as nn
>>> pe = nn.SinusoidalEmbedding(num_positions=512, embedding_dim=64)
>>> table = pe()
>>> table.shape
(512, 64)