gpt2_small

→GPT2Model

gpt2_small(pretrained: bool | str = False, weights: GPT2SmallWeights | None = None, overrides: object = {})

source edit

Construct a GPT-2 small (124M) decoder trunk.

Smallest of the four canonical OpenAI variants from Radford, Wu, Child, Luan, Amodei, and Sutskever, 2019: $L=12$ transformer blocks, $H=768$ hidden, $A=12$ attention heads, intermediate width 3072. Roughly 124M parameters (historically released as the "117M" file before a re-count). This is the default entry point for GPT-2 research and the parameter-count match for GPT-1.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(GPT2SmallWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional GPT2Config field overrides forwarded into the underlying config.

Returns

GPT2Model

Decoder trunk configured with the GPT-2 small size and any overrides.

Notes

Reference: Radford, Wu, Child, Luan, Amodei, and Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI Technical Report, 2019.

Examples

>>> import lucid
>>> from lucid.models.text.gpt2 import gpt2_small
>>> model = gpt2_small().eval()
>>> input_ids = lucid.tensor([[15496, 11, 995, 13]])   # "Hello, world."
>>> out = model(input_ids)
>>> out.last_hidden_state.shape   # (1, 4, 768)
(1, 4, 768)

Used by 1

lucid.models.text.gpt2

>>> import lucid >>> from lucid.models.text.gpt2 import gpt2_small >>> model = gpt2_small().eval() >>> input_ids = lucid.tensor([[15496, 11, 995, 13]]) # "Hello, world." >>> out = model(input_ids) >>> out.last_hidden_state.shape # (1, 4, 768) (1, 4, 768)