class

RoFormerModel

extendsPretrainedModel

RoFormerModel(config: RoFormerConfig)

source edit

Bare RoFormer encoder trunk with rotary position embeddings.

Architecturally a BERT-base encoder (12 layers, 768 hidden, 12 heads, intermediate width 3072) where the absolute learned position embedding is removed and rotary position embedding (RoPE) is applied to the query and key vectors inside every self-attention layer. RoPE injects relative-position information multiplicatively inside attention so the dot product $\langle R_m q, R_n k \rangle$ depends only on the offset $m - n$ — granting better length extrapolation and composing naturally with linear attention.

Parameters

configRoFormerConfig

Hyperparameters controlling vocabulary, depth, width, head count, and the RoPE rotary_base. The per-head dimension hidden_size / num_attention_heads must be even (enforced by RoFormerConfig).

Attributes

embeddingsnn.Module

Token + token-type embedding block followed by LayerNorm and dropout. Notably no learned position embedding.

rotaryModule

Pre-computed interleaved-layout (cos, sin) tables of shape (max_position_embeddings, head_dim), sliced to the actual sequence length at each forward. Uses the original RoPE pairing (x_2i, x_2i+1) (each frequency repeated twice) rather than the shared half-split layout, matching the RoFormer reference checkpoints.

encodernn.Module

Stack of config.num_hidden_layers rotary-attention transformer layers.

poolernn.Module

Dense + tanh projection of the first token's hidden state used by sentence-level classification heads.

Notes

Reference: Su, Lu, Pan, Murtadha, Wen, and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024, article 127063 (arXiv:2104.09864).

Rotary rotation per $(\cos\theta_i, \sin\theta_i)$ pair, with $\theta_i = \mathrm{base}^{-2i/d_{\text{head}}}$ :

R_\theta = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}.

Applied independently to every $(x_{2i}, x_{2i+1})$ pair of $Q$ and $K$ so $q_m^\top k_n$ becomes a function of $m - n$ only.

Examples

>>> import lucid
>>> from lucid.models.text.roformer import RoFormerConfig, RoFormerModel
>>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128,
...                      num_attention_heads=2, intermediate_size=512)
>>> model = RoFormerModel(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.last_hidden_state.shape, out.pooler_output.shape
((1, 4, 128), (1, 128))

Used by 2

Constructors

dunder

init

→None

__init__(config: RoFormerConfig)

source edit

Instance methods

forward

→BaseModelOutputWithPooling

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None)

source edit

get_input_embeddings

→nn.Module

get_input_embeddings()

source edit

set_input_embeddings

→None

set_input_embeddings(value: nn.Module)

source edit

>>> import lucid >>> from lucid.models.text.roformer import RoFormerConfig, RoFormerModel >>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = RoFormerModel(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.last_hidden_state.shape, out.pooler_output.shape ((1, 4, 128), (1, 128))