transformer_base_cls

→TransformerForSequenceClassification

transformer_base_cls(pretrained: bool = False, overrides: object = {})

source edit

Construct a Transformer "base" with a sequence-classification head.

Same trunk as transformer_base (N=6, d_model=512, h=8); only the encoder half is used in forward. The first source-side token's encoder hidden state is dropout-regularised and projected through a linear of output width config.num_labels. Pattern mirrors BERTForSequenceClassification and is appropriate for GLUE-style fine-tunes on the encoder-only half of the Vaswani trunk.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

**overridesobject= {}

Optional TransformerConfig field overrides forwarded into the underlying config. Pass num_labels=N to set the number of classes (default 2).

Returns

TransformerForSequenceClassification

Encoder-only Transformer trunk wrapped with a pooled classifier head.

Notes

Reference: Vaswani et al., "Attention Is All You Need", NeurIPS, 2017.

Examples

>>> import lucid
>>> from lucid.models.text.transformer import transformer_base_cls
>>> model = transformer_base_cls(num_labels=3).eval()
>>> input_ids = lucid.tensor([[1, 234, 567, 2]])
>>> out = model(input_ids)
>>> out.logits.shape   # (1, num_labels=3)
(1, 3)

Used by 1

lucid.models.text.transformer