roformer_qa(pretrained: bool | str = False, weights: RoFormerWeights | None = None, overrides: object = {})Construct a RoFormer model with an extractive span head.
A two-way linear over each position, scoring it as the start and the end of the answer span.
Model Size
Parameters
pretrainedbool or str= FalseEncoder-weight selector.
False → fully random init; True
→ loads the roformer encoder
(RoFormerWeights.DEFAULT) into the .roformer trunk.
The span head is always randomly initialised — no
SQuAD-fine-tuned RoFormer checkpoint ships here.Explicit encoder-weights enum member; takes precedence over
pretrained.**overridesobject= {}Optional
RoFormerConfig field overrides.Returns
RoFormerForQuestionAnsweringRoFormer trunk wrapped with the span head.
Notes
Reference: Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).
Examples
>>> import lucid
>>> from lucid.models.text.roformer import roformer_qa
>>> model = roformer_qa(vocab_size=99, hidden_size=32,
... num_hidden_layers=2, num_attention_heads=2,
... intermediate_size=64).eval()
>>> out = model(lucid.tensor([[1, 2, 3, 4]]))
>>> out.start_logits.shape, out.end_logits.shape
((1, 4), (1, 4))