SinusoidalEmbedding
ModuleSinusoidalEmbedding(num_positions: int, embedding_dim: int, base: float = 10000.0)Fixed 1-D sinusoidal positional encoding (Vaswani et al., 2017).
Carries no learnable parameters — the encoding table is precomputed once
in __init__ and stored as a non-persistent buffer.
Args:
num_positions: Maximum sequence length supported.
embedding_dim: Per-position embedding size; must be even.
base: Frequency base θ_0 in the sin/cos formula. 10000 per
Vaswani 2017; some long-context models use larger.
Forward:
Returns the full (num_positions, embedding_dim) table. Callers
typically slice it with pe()[:seq_len] and broadcast against
their batched hidden states.
Notes
The encoding table is materialised once in __init__ and registered
as a non-persistent buffer; it is therefore not learnable and is not
saved in state_dict (but moves with .to(device=...)). Prefer
the module form when the same maximum sequence length is reused across
many forward passes; use the functional
lucid.nn.functional.sinusoidal_embedding form for one-shot or
variable-length encodings where caching is wasteful. See that function
for the underlying sin/cos formula.
Examples
>>> import lucid.nn as nn
>>> pe = nn.SinusoidalEmbedding(num_positions=512, embedding_dim=64)
>>> table = pe()
>>> table.shape
(512, 64)