transformer_large

→TransformerModel

transformer_large(pretrained: bool = False, overrides: object = {})

source edit

Construct a Vaswani-style Transformer "big" encoder-decoder.

Larger of the two original variants from Vaswani et al., 2017, Table 3 ("big" row): $N=6$ encoder + $N=6$ decoder layers, $d_{\text{model}} = 1024$ , $h = 16$ heads, $d_{\text{ff}} = 4096$ , dropout 0.3. Roughly 213M parameters — pushed BLEU on WMT 2014 En-De to 28.4 / En-Fr to 41.8 at release time.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

**overridesobject= {}

Optional TransformerConfig field overrides forwarded into the underlying config.

Returns

TransformerModel

Encoder-decoder trunk configured with the "big" size and any overrides.

Notes

Reference: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin, "Attention Is All You Need", NeurIPS, 2017 (arXiv:1706.03762), Table 3.

Examples

>>> import lucid
>>> from lucid.models.text.transformer import transformer_large
>>> model = transformer_large().eval()
>>> src = lucid.tensor([[1, 234, 567, 2]])
>>> tgt = lucid.tensor([[1, 100, 200]])
>>> out = model(src, decoder_input_ids=tgt)
>>> out.logits.shape   # (B=1, T_tgt=3, d_model=1024)
(1, 3, 1024)

Used by 1

lucid.models.text.transformer

>>> import lucid >>> from lucid.models.text.transformer import transformer_large >>> model = transformer_large().eval() >>> src = lucid.tensor([[1, 234, 567, 2]]) >>> tgt = lucid.tensor([[1, 100, 200]]) >>> out = model(src, decoder_input_ids=tgt) >>> out.logits.shape # (B=1, T_tgt=3, d_model=1024) (1, 3, 1024)