transformer_base_seq2seq

→TransformerForSeq2SeqLM

transformer_base_seq2seq(pretrained: bool = False, overrides: object = {})

source edit

Construct a Transformer "base" with a tied seq2seq-LM head.

Same trunk as transformer_base (N=6, d_model=512, h=8), augmented with a linear LM head whose weight is tied to the target-side embedding when config.tie_word_embeddings is True. Provides a .generate(input_ids) method for greedy seq2seq decoding (translation / summarisation inference).

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

**overridesobject= {}

Optional TransformerConfig field overrides forwarded into the underlying config.

Returns

TransformerForSeq2SeqLM

Encoder-decoder trunk wrapped with the tied LM head.

Notes

Reference: Vaswani et al., "Attention Is All You Need", NeurIPS, 2017.

Training loss when labels is supplied is the standard seq2seq cross-entropy:

\mathcal{L}_{\mathrm{S2S}} = -\frac{1}{BT} \sum_{b, t} \log p_\theta(y_{b, t} \mid x_b, y_{b, < t}),

with positions labelled -100 excluded.

Examples

>>> import lucid
>>> from lucid.models.text.transformer import transformer_base_seq2seq
>>> model = transformer_base_seq2seq().eval()
>>> src = lucid.tensor([[1, 234, 567, 2]])
>>> tgt = lucid.tensor([[1, 100, 200]])
>>> out = model(src, decoder_input_ids=tgt)
>>> out.logits.shape   # (B=1, T_tgt=3, vocab=37000)
(1, 3, 37000)

Used by 1

lucid.models.text.transformer

>>> import lucid >>> from lucid.models.text.transformer import transformer_base_seq2seq >>> model = transformer_base_seq2seq().eval() >>> src = lucid.tensor([[1, 234, 567, 2]]) >>> tgt = lucid.tensor([[1, 100, 200]]) >>> out = model(src, decoder_input_ids=tgt) >>> out.logits.shape # (B=1, T_tgt=3, vocab=37000) (1, 3, 37000)