GPT-1 + LM head + multiple-choice classification head.
Implements the Radford 2018 §3.3 multi-task fine-tuning recipe. Used for tasks like RACE / Story Cloze where each example presents a prompt followed by candidate completions; the model is jointly trained with the standard LM objective on each choice and a -way classification loss picking the correct completion.
Inputs are 3-D: input_ids of shape (N, C, L) flattens to
(N * C, L) for the trunk and reshapes back to (N, C, ...) at
the heads. mc_token_ids of shape (N, C) indexes the per-choice
pooling position (typically the trailing CLF token of each completion).
Parameters
configGPTConfigconfig.tie_word_embeddings (default True)
ties the LM head weight to the input embedding matrix.Attributes
Notes
Reference: Radford, Narasimhan, Salimans, and Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018, §3.3.
Combined loss when both objectives are supplied:
Examples
>>> import lucid
>>> from lucid.models.text.gpt import GPTConfig, GPTDoubleHeadsModel
>>> cfg = GPTConfig(num_hidden_layers=2, hidden_size=64,
... num_attention_heads=2, intermediate_size=256)
>>> model = GPTDoubleHeadsModel(cfg).eval()
>>> input_ids = lucid.tensor([[[101, 7592, 102],
... [101, 2088, 102]]]) # (N=1, C=2, L=3)
>>> mc_ids = lucid.tensor([[2, 2]]).long()
>>> out = model(input_ids, mc_token_ids=mc_ids)
>>> out.mc_logits.shape # (N=1, C=2)
(1, 2)Used by 1
Constructors
1Instance methods
1forward(input_ids: Tensor, mc_token_ids: Tensor, attention_mask: Tensor | None = None, labels: Tensor | None = None, mc_labels: Tensor | None = None)