GPT-2 + LM head + multiple-choice classification head.
Identical contract to GPTDoubleHeadsModel but with GPT-2's
pre-LN trunk and transformer.wte parameter naming so checkpoints
port directly. Used for tasks like RACE / Story Cloze where each
example presents a prompt followed by C candidate completions;
the model is jointly trained with the standard LM objective on each
choice and a C-way classification loss picking the correct one.
Inputs are 3-D: input_ids of shape (N, C, L) flattens to
(N * C, L) for the trunk and reshapes back to (N, C, ...) at
the heads. mc_token_ids of shape (N, C) indexes the per-choice
pooling position (typically the trailing CLF token of each completion).
Parameters
configGPT2Configconfig.tie_word_embeddings (default True)
ties the LM head weight to the wte embedding matrix.Attributes
Notes
Reference: Radford et al., "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019; multiple-choice head follows the GPT-1 fine-tuning recipe (Radford 2018 §3.3).
The combined loss when both objectives are supplied is
where is a softmax cross-entropy
over the C candidate completions.
Examples
>>> import lucid
>>> from lucid.models.text.gpt2 import GPT2Config, GPT2DoubleHeadsModel
>>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=64,
... num_attention_heads=2, intermediate_size=256)
>>> model = GPT2DoubleHeadsModel(cfg).eval()
>>> input_ids = lucid.tensor([[[15496, 11, 995, 13],
... [15496, 11, 1820, 13]]]) # (N=1, C=2, L=4)
>>> mc_ids = lucid.tensor([[3, 3]]).long()
>>> out = model(input_ids, mc_token_ids=mc_ids)
>>> out.mc_logits.shape # (N=1, C=2)
(1, 2)Used by 1
Constructors
1Instance methods
1forward(input_ids: Tensor, mc_token_ids: Tensor, attention_mask: Tensor | None = None, labels: Tensor | None = None, mc_labels: Tensor | None = None)