data
TransformerModelOutput
extends
ModelOutputTransformerModelOutput(last_hidden_state: Tensor, encoder_last_hidden_state: Tensor)Output of the bare encoder-decoder trunk (no LM head).
Attributes
Notes
The trunk used to return Seq2SeqLMOutput with the decoder
hidden states parked in its logits field. Those are activations
of width d_model, not scores over a vocabulary, and a caller that
softmaxed them — as the name invites — got a distribution over
hidden units. TransformerForSeq2SeqLM is the model that
produces real logits.
Examples
>>> import lucid
>>> from lucid.models.text.transformer._model import (
... TransformerModelOutput,
... )
>>> out = TransformerModelOutput(
... last_hidden_state=lucid.zeros(1, 32, 512),
... encoder_last_hidden_state=lucid.zeros(1, 128, 512),
... )
>>> out.last_hidden_state.shape, out.encoder_last_hidden_state.shape
((1, 32, 512), (1, 128, 512))
TransformerModel(cfg)(input_ids, decoder_input_ids) returns this
type. The two lengths differ because the decoder is partway through a
sequence the encoder has already read in full.