data
GPTDoubleHeadsOutput
extends
ModelOutputGPTDoubleHeadsOutput(lm_logits: Tensor, mc_logits: Tensor, loss: Tensor | None = None, lm_loss: Tensor | None = None, mc_loss: Tensor | None = None)Joint LM + multiple-choice output for GPTDoubleHeadsModel.
Aggregates the per-choice causal-LM logits, per-choice classification logits, and the optional per-objective and combined losses produced when training Radford-style multiple-choice fine-tunes.
Parameters
lm_logitsTensorLM head logits of shape
(N, C, L, vocab_size) — one
distribution per token of every batch x choice pair.mc_logitsTensorMultiple-choice head logits of shape
(N, C) — one scalar per
candidate completion.Sum of
lm_loss and mc_loss when both are available;
otherwise the single available loss, or None.Shift-cross-entropy on the LM head when
labels is supplied.Cross-entropy on the multiple-choice head when
mc_labels is
supplied.Notes
Reference: Radford, Narasimhan, Salimans, and Sutskever, "Improving Language Understanding by Generative Pre-Training", OpenAI Technical Report, 2018, §3.3.
Examples
>>> import lucid
>>> from lucid.models.text.gpt import GPTConfig, GPTDoubleHeadsModel
>>> cfg = GPTConfig(num_hidden_layers=2, hidden_size=64,
... num_attention_heads=2, intermediate_size=256)
>>> model = GPTDoubleHeadsModel(cfg).eval()
>>> input_ids = lucid.tensor([[[101, 7592, 102]]]) # (N=1, C=1, L=3)
>>> mc_ids = lucid.tensor([[2]]).long()
>>> out = model(input_ids, mc_token_ids=mc_ids)
>>> out.lm_logits.shape, out.mc_logits.shape
((1, 1, 3, 40478), (1, 1))