roformer_mlm(pretrained: bool | str = False, weights: RoFormerMLMWeights | None = None, overrides: object = {})Construct a RoFormer model with a tied masked-LM head.
Same trunk as roformer (L=12, H=768, A=12), augmented with the
BERT-style two-layer projection mapping each hidden state to vocabulary
logits via a decoder whose weight matrix is tied to the input
word_embeddings table when config.tie_word_embeddings is True.
Use for MLM pre-training when RoPE relative-position semantics are
preferred to absolute position embeddings.
Model Size
Parameters
pretrainedbool= FalseReserved for future weight registration; currently a no-op.
weights(RoFormerMLMWeights, optional, keyword - only)= NoneExplicit weights enum member; takes precedence over
pretrained.**overridesobject= {}Optional
RoFormerConfig field overrides forwarded into the
underlying config.Returns
RoFormerForMaskedLMRoFormer trunk wrapped with the tied MLM head.
Notes
Reference: Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).
Examples
>>> import lucid
>>> from lucid.models.text.roformer import roformer_mlm
>>> model = roformer_mlm().eval()
>>> input_ids = lucid.tensor([[101, 7592, 103, 102]]) # [CLS] hello [MASK] [SEP]
>>> out = model(input_ids)
>>> out.logits.shape # (1, 4, vocab=50000)
(1, 4, 50000)