class

GPT2ForSequenceClassification

extendsPretrainedModel

GPT2ForSequenceClassification(config: GPT2Config)

source edit

GPT-2 with a last-token sequence-classification head.

Wraps GPT2Model with a dropout-regularised linear classifier operating on the rightmost non-pad token's hidden state — the canonical decoder-style pooling. Unlike BERT (CLS at position 0), decoder LMs only have a meaningful summary at the last attended position since every preceding position has attended to a strict prefix.

Parameters

configGPT2Config

Hyperparameters. config.num_labels sets the output dimension; config.classifier_dropout (falling back to hidden_dropout) sets the dropout applied before the linear.

Attributes

transformerGPT2Model

Underlying decoder trunk.

dropoutnn.Dropout

Dropout layer applied to the pooled last-token embedding.

classifiernn.Linear

Final linear of shape (hidden_size, num_labels) producing per-class logits.

Notes

Reference: Radford et al., "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019.

Pooling picks index $t^\star = \sum_t \mathbf{1}[\mathrm{mask}_t] - 1$ (the rightmost real token) per row, then computes

z = W_{\mathrm{cls}}\, \mathrm{Dropout}(h_{t^\star}).

When labels is supplied, cross-entropy over num_labels classes is computed and exposed as output.loss.

Examples

>>> import lucid
>>> from lucid.models.text.gpt2 import GPT2Config, GPT2ForSequenceClassification
>>> cfg = GPT2Config(num_labels=3, num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = GPT2ForSequenceClassification(cfg).eval()
>>> input_ids = lucid.tensor([[15496, 11, 995, 13]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, num_labels=3)
(1, 3)

Used by 2

Constructors

dunder

init

→None

__init__(config: GPT2Config)

source edit

Instance methods

forward

→MaskedLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.gpt2 import GPT2Config, GPT2ForSequenceClassification >>> cfg = GPT2Config(num_labels=3, num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = GPT2ForSequenceClassification(cfg).eval() >>> input_ids = lucid.tensor([[15496, 11, 995, 13]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, num_labels=3) (1, 3)