transformer_base

→TransformerModel

transformer_base(pretrained: bool = False, overrides: object = {})

source edit

Construct a Vaswani-style Transformer "base" encoder-decoder.

Canonical seq2seq architecture from Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin, 2017, Table 3 ("base" row): $N=6$ encoder + $N=6$ decoder layers, $d_{\text{model}} = 512$ , $h = 8$ heads, $d_{\text{ff}} = 2048$ , dropout 0.1. Roughly 65M parameters — the model that achieved SOTA on WMT 2014 En-De / En-Fr at a fraction of the training cost of prior recurrent seq2seq systems.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

**overridesobject= {}

Optional TransformerConfig field overrides (e.g. vocab_size=..., decoder_vocab_size=..., share_embeddings=True) forwarded into the underlying config.

Returns

TransformerModel

Encoder-decoder trunk configured with the "base" size and any overrides.

Notes

Reference: Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin, "Attention Is All You Need", NeurIPS, 2017 (arXiv:1706.03762), Table 3.

Multi-head attention:

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left( \frac{Q K^{\top}}{\sqrt{d_k}} \right) V.

Examples

>>> import lucid
>>> from lucid.models.text.transformer import transformer_base
>>> model = transformer_base().eval()
>>> src = lucid.tensor([[1, 234, 567, 2]])
>>> tgt = lucid.tensor([[1, 100, 200]])
>>> out = model(src, decoder_input_ids=tgt)
>>> out.logits.shape   # (B=1, T_tgt=3, d_model=512)
(1, 3, 512)

Used by 1

lucid.models.text.transformer

>>> import lucid >>> from lucid.models.text.transformer import transformer_base >>> model = transformer_base().eval() >>> src = lucid.tensor([[1, 234, 567, 2]]) >>> tgt = lucid.tensor([[1, 100, 200]]) >>> out = model(src, decoder_input_ids=tgt) >>> out.logits.shape # (B=1, T_tgt=3, d_model=512) (1, 3, 512)