data
TransformerConfig
extends
LanguageModelConfigTransformerConfig(vocab_size: int = 37000, hidden_size: int = 512, num_hidden_layers: int = 6, num_attention_heads: int = 8, intermediate_size: int = 2048, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 5000, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, decoder_vocab_size: int | None = None, num_decoder_layers: int = 6, share_embeddings: bool = False, num_labels: int = 2, classifier_dropout: float | None = None)Configuration for every Vaswani-style Transformer variant.
The bare paper specifies a single 65 537-token shared BPE vocabulary for
WMT En-De and separate vocabularies for some tasks; we make this
configurable via decoder_vocab_size / share_embeddings so callers
can match whichever checkpoint they're porting.
Used by 3
Constructors
1dunder
__init__
→None__init__(vocab_size: int = 37000, hidden_size: int = 512, num_hidden_layers: int = 6, num_attention_heads: int = 8, intermediate_size: int = 2048, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 5000, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, decoder_vocab_size: int | None = None, num_decoder_layers: int = 6, share_embeddings: bool = False, num_labels: int = 2, classifier_dropout: float | None = None)