class

RoFormerForMaskedLM

extendsPretrainedModelMaskedLMMixin

RoFormerForMaskedLM(config: RoFormerConfig)

source edit

RoFormer with a tied masked-language-modeling head.

Wraps RoFormerModel with the BERT-style two-layer projection (dense + activation + LayerNorm) mapping each hidden state to vocabulary logits. The decoder weight is tied to the input word_embeddings table when config.tie_word_embeddings is True. Use for MLM pre-training when RoPE-based relative position information is preferred to absolute position embeddings.

Parameters

configRoFormerConfig

Hyperparameters. config.tie_word_embeddings (default True) controls whether the MLM decoder weight is bound to the input embedding matrix.

Attributes

roformerRoFormerModel

Underlying RoPE encoder trunk.

clsnn.Module

Masked-LM prediction head — dense + LayerNorm transform plus a decoder of shape (hidden_size, vocab_size).

Notes

Reference: Su, Lu, Pan, Murtadha, Wen, and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).

When labels is provided, the loss is the standard masked cross-entropy

\mathcal{L}_{\mathrm{MLM}} = -\frac{1}{|M|} \sum_{i \in M} \log p_\theta(x_i \mid x_{\setminus M}),

over the set $M$ of masked positions (positions whose label equals -100 are excluded).

Examples

>>> import lucid
>>> from lucid.models.text.roformer import RoFormerConfig, RoFormerForMaskedLM
>>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128,
...                      num_attention_heads=2, intermediate_size=512)
>>> model = RoFormerForMaskedLM(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 103, 102]])   # [CLS] hello [MASK] [SEP]
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, T=4, V=50000)
(1, 4, 50000)

Used by 2

Constructors

dunder

init

→None

__init__(config: RoFormerConfig)

source edit

Instance methods

forward

→MaskedLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.roformer import RoFormerConfig, RoFormerForMaskedLM >>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = RoFormerForMaskedLM(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 103, 102]]) # [CLS] hello [MASK] [SEP] >>> out = model(input_ids) >>> out.logits.shape # (B=1, T=4, V=50000) (1, 4, 50000)