class

RoFormerForMultipleChoice

extendsPretrainedModel

RoFormerForMultipleChoice(config: RoFormerConfig)

source edit

RoFormer with a per-choice multiple-choice classification head.

Wraps RoFormerModel with a per-choice CLS classifier. Input (N, C, L) flattens to (N*C, L) for the encoder; the pooled CLS embedding of each choice is projected to a scalar and the resulting (N, C) matrix is the per-choice logit. Used for fine-tunes on SWAG / RACE-style commonsense and reading-comprehension tasks.

Parameters

configRoFormerConfig

Hyperparameters. config.classifier_dropout (falling back to hidden_dropout) sets the dropout applied before the linear.

Attributes

roformerRoFormerModel

Underlying RoPE encoder trunk.

dropoutnn.Dropout

Dropout layer applied to each choice's pooled embedding.

classifiernn.Linear

Per-choice scalar projection of shape (hidden_size, 1).

Notes

Reference: Su, Lu, Pan, Murtadha, Wen, and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).

When labels is supplied the loss is the softmax cross-entropy over the C candidates:

\mathcal{L}_{\mathrm{MC}} = -\frac{1}{N} \sum_{n=1}^{N} \log \mathrm{softmax}(z_n)_{y_n}.

Examples

>>> import lucid
>>> from lucid.models.text.roformer import (
...     RoFormerConfig, RoFormerForMultipleChoice,
... )
>>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128,
...                      num_attention_heads=2, intermediate_size=512)
>>> model = RoFormerForMultipleChoice(cfg).eval()
>>> input_ids = lucid.tensor([[[101, 7592, 102], [101, 2088, 102]]])    # (N=1, C=2, L=3)
>>> out = model(input_ids)
>>> out.logits.shape   # (N=1, C=2)
(1, 2)

Used by 1

lucid.models.text.roformer

Constructors

dunder

init

→None

__init__(config: RoFormerConfig)

source edit

Instance methods

forward

→MaskedLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.roformer import ( ... RoFormerConfig, RoFormerForMultipleChoice, ... ) >>> cfg = RoFormerConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = RoFormerForMultipleChoice(cfg).eval() >>> input_ids = lucid.tensor([[[101, 7592, 102], [101, 2088, 102]]]) # (N=1, C=2, L=3) >>> out = model(input_ids) >>> out.logits.shape # (N=1, C=2) (1, 2)