gpt_cls

→GPTForSequenceClassification

gpt_cls(pretrained: bool | str = False, weights: GPTWeights | None = None, overrides: object = {})

source edit

Construct a GPT-1 model with a last-token sequence-classification head.

Same trunk as gpt (L=12, H=768, A=12), augmented with a linear classifier on the last non-pad token's hidden state — the canonical decoder-style pooling strategy. Use for GLUE-style classification when you specifically want autoregressive features rather than bidirectional BERT-style ones.

Model Size

Parameters

pretrainedbool or str= False

Encoder-weight selector. False → fully random init; True → loads the pretrained gpt decoder trunk (GPTWeights.DEFAULT) into the .transformer submodule; a tag string selects a specific encoder checkpoint. The classifier head is always randomly initialised (fine-tuning starting point — no GLUE-fine-tuned head ships).

weights(GPTWeights, optional, keyword - only)= None

Explicit encoder-weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional GPTConfig field overrides forwarded into the underlying config. Pass num_labels=N to set the number of output classes (default 2). Overrides that change the trunk shape are incompatible with loading pretrained encoder weights.

Returns

GPTForSequenceClassification

GPT-1 trunk wrapped with the last-token classifier head (encoder pretrained when requested; head random).

Notes

Reference: Radford et al., "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018, §3.3.

Pooling picks index $t^\star = \sum_t \mathbf{1}[\mathrm{mask}_t] - 1$ (i.e. the rightmost real token) per row, then applies dropout + linear:

z = W_{\mathrm{cls}}\, \mathrm{Dropout}(h_{t^\star}) + b_{\mathrm{cls}}.

Examples

>>> import lucid
>>> from lucid.models.text.gpt import gpt_cls
>>> model = gpt_cls(num_labels=3).eval()
>>> input_ids = lucid.tensor([[101, 7592, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, num_labels=3)
(1, 3)

Used by 1

lucid.models.text.gpt

>>> import lucid >>> from lucid.models.text.gpt import gpt_cls >>> model = gpt_cls(num_labels=3).eval() >>> input_ids = lucid.tensor([[101, 7592, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, num_labels=3) (1, 3)