class

RoFormerForSequenceClassification

extendsPretrainedModel

RoFormerForSequenceClassification(config: RoFormerConfig)

source edit

RoFormer with a pooled-CLS sequence-classification head.

Wraps RoFormerModel with a dropout-regularised linear classifier operating on the pooled first-token embedding. Pattern mirrors BERTForSequenceClassification; use for GLUE-style fine-tunes when the RoPE-based relative position bias is preferred.

Parameters

configRoFormerConfig

Hyperparameters. config.num_labels sets the output dimension; config.classifier_dropout (falling back to hidden_dropout) sets the dropout applied before the linear.

Attributes

roformerRoFormerModel

Underlying RoPE encoder trunk.

dropoutnn.Dropout

Dropout layer applied to the pooled first-token embedding.

classifiernn.Linear

Final linear of shape (hidden_size, num_labels) producing per-class logits.

Notes

Reference: Su, Lu, Pan, Murtadha, Wen, and Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", Neurocomputing, vol. 568, 2024 (arXiv:2104.09864).

Loss (when labels is provided) is the standard softmax cross-entropy:

\mathcal{L} = -\frac{1}{B} \sum_{b=1}^{B} \log p_\theta(y_b \mid x_b).

Examples

>>> import lucid
>>> from lucid.models.text.roformer import (
...     RoFormerConfig, RoFormerForSequenceClassification,
... )
>>> cfg = RoFormerConfig(num_labels=3, num_hidden_layers=2,
...                      hidden_size=128, num_attention_heads=2,
...                      intermediate_size=512)
>>> model = RoFormerForSequenceClassification(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, num_labels=3)
(1, 3)

Used by 2

Constructors

dunder

init

→None

__init__(config: RoFormerConfig)

source edit

Instance methods

forward

→MaskedLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.roformer import ( ... RoFormerConfig, RoFormerForSequenceClassification, ... ) >>> cfg = RoFormerConfig(num_labels=3, num_hidden_layers=2, ... hidden_size=128, num_attention_heads=2, ... intermediate_size=512) >>> model = RoFormerForSequenceClassification(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, num_labels=3) (1, 3)