data
GPT2DoubleHeadsOutput
extends
ModelOutputGPT2DoubleHeadsOutput(lm_logits: Tensor, mc_logits: Tensor, loss: Tensor | None = None, lm_loss: Tensor | None = None, mc_loss: Tensor | None = None)Joint LM + multiple-choice output for GPT2DoubleHeadsModel.
Aggregates the per-choice causal-LM logits, the per-choice multiple- choice classification logits, and the optional per-objective and combined losses produced when training Radford-style multiple-choice fine-tunes (Story Cloze, RACE).
Parameters
lm_logitsTensorLM head logits of shape
(N, C, L, vocab_size) — one
distribution per token of every batch x choice pair.mc_logitsTensorMultiple-choice head logits of shape
(N, C) — one scalar per
candidate completion.Sum of
lm_loss and mc_loss when both are available;
otherwise the single available loss, or None.Shift-cross-entropy on the LM head when
labels is supplied.Cross-entropy on the multiple-choice head when
mc_labels is
supplied.Notes
Reference: Radford et al., "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019.
Examples
>>> import lucid
>>> from lucid.models.text.gpt2 import GPT2Config, GPT2DoubleHeadsModel
>>> cfg = GPT2Config(num_hidden_layers=2, hidden_size=64,
... num_attention_heads=2, intermediate_size=256)
>>> model = GPT2DoubleHeadsModel(cfg).eval()
>>> input_ids = lucid.tensor([[[15496, 11, 995, 13]]]) # (N=1, C=1, L=4)
>>> mc_ids = lucid.tensor([[3]]).long()
>>> out = model(input_ids, mc_token_ids=mc_ids)
>>> out.lm_logits.shape, out.mc_logits.shape
((1, 1, 4, 50257), (1, 1))