roformer

→RoFormerModel

roformer(pretrained: bool | str = False, weights: RoFormerWeights | None = None, overrides: object = {})

source edit

Construct a RoFormer encoder trunk.

Canonical RoFormer architecture from Su, Lu, Pan, Murtadha, Wen, and Liu, 2024: $L=12$ transformer layers, $H=768$ hidden, $A=12$ attention heads, intermediate width 3072 — same shape as BERT-base, but with absolute learned position embeddings replaced by rotary position embedding (RoPE) applied to $Q$ and $K$ inside every self-attention layer. Max sequence length is extended from 512 to 1536 to exercise RoPE's length-extrapolation property.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(RoFormerWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional RoFormerConfig field overrides (e.g. vocab_size=..., rotary_base=..., max_position_embeddings=...) forwarded into the underlying config.

Returns

RoFormerModel

Encoder trunk configured with the paper defaults plus any overrides.

Notes

Reference: Su, Lu, Pan, Murtadha, Wen, and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024, article 127063 (arXiv:2104.09864).

Rotary rotation matrix per $(\cos\theta_i, \sin\theta_i)$ pair:

R_\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}.

Examples

>>> import lucid
>>> from lucid.models.text.roformer import roformer
>>> model = roformer().eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.last_hidden_state.shape, out.pooler_output.shape
((1, 4, 768), (1, 768))

Used by 1

lucid.models.text.roformer

>>> import lucid >>> from lucid.models.text.roformer import roformer >>> model = roformer().eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.last_hidden_state.shape, out.pooler_output.shape ((1, 4, 768), (1, 768))