data
GPT2Config
extends
LanguageModelConfigGPT2Config(vocab_size: int = 50257, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu_new', max_position_embeddings: int = 1024, pad_token_id: int | None = None, bos_token_id: int | None = 50256, eos_token_id: int | None = 50256, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-05, tie_word_embeddings: bool = True, use_cache: bool = True, num_labels: int = 2, classifier_dropout: float | None = None, scale_residual_init: bool = True)Configuration for every GPT-2 variant.
Defaults match the 124M-parameter "small" checkpoint released in 2019
(the file historically known as 117M). Larger sizes are reached by
overriding hidden_size / num_hidden_layers / num_attention_heads
— see lucid.models.text.gpt2._pretrained for the canonical table.
Used by 3
Constructors
1dunder
__init__
→None__init__(vocab_size: int = 50257, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu_new', max_position_embeddings: int = 1024, pad_token_id: int | None = None, bos_token_id: int | None = 50256, eos_token_id: int | None = 50256, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-05, tie_word_embeddings: bool = True, use_cache: bool = True, num_labels: int = 2, classifier_dropout: float | None = None, scale_residual_init: bool = True)