roformer_multiple_choice(pretrained: bool | str = False, weights: RoFormerWeights | None = None, overrides: object = {})Construct a RoFormer model with a multiple-choice head.
Scores each candidate independently and softmaxes across them, so the
input is (N, C, L) — batch, choices, tokens — rather than the
(N, L) every other head here takes. The choices are flattened
into the batch, run through the shared trunk, and reshaped back to
(N, C) logits.
Model Size
Parameters
pretrainedbool or str= FalseEncoder-weight selector.
False → fully random init; True
→ loads the roformer encoder
(RoFormerWeights.DEFAULT) into the .roformer trunk.
The scoring head is always randomly initialised.Explicit encoder-weights enum member; takes precedence over
pretrained.**overridesobject= {}Optional
RoFormerConfig field overrides. num_labels
does not apply — the head is a single scalar per choice, and the
choice count comes from the input's second axis.Returns
RoFormerForMultipleChoiceRoFormer trunk wrapped with the per-choice scorer.
Notes
Reference: Su et al., "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).
The head reads the last token's hidden state, not a pooled [CLS]
— see RoFormerForMultipleChoice for why.
Examples
>>> import lucid
>>> from lucid.models.text.roformer import roformer_multiple_choice
>>> model = roformer_multiple_choice(vocab_size=99, hidden_size=32,
... num_hidden_layers=2,
... num_attention_heads=2,
... intermediate_size=64).eval()
>>> ids = lucid.tensor([[[1, 2, 3, 4], [1, 2, 3, 5]]]) # (N=1, C=2, L=4)
>>> out = model(ids)
>>> out.logits.shape # (N=1, C=2)
(1, 2)