data
TransformerConfig
extends
LanguageModelConfigTransformerConfig(vocab_size: int = 37000, hidden_size: int = 512, num_hidden_layers: int = 6, num_attention_heads: int = 8, intermediate_size: int = 2048, hidden_act: TextActivation = 'relu', max_position_embeddings: int = 5000, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, label_smoothing: float = 0.1, decoder_vocab_size: int | None = None, num_decoder_layers: int = 6, share_embeddings: bool = True, num_labels: int = 2, classifier_dropout: float | None = None, encoder_only: bool = False)Configuration for every Vaswani-style Transformer variant.
The paper shares one BPE vocabulary — and one embedding matrix — across
source, target and the pre-softmax projection (§3.4), which is the
default here. decoder_vocab_size / share_embeddings stay
configurable so callers porting a checkpoint with split vocabularies can
turn sharing off.
Examples
>>> from lucid.models.text.transformer._config import TransformerConfig
>>> cfg = TransformerConfig()
>>> cfg.model_type
'transformer'
>>> cfg.hidden_size, cfg.vocab_size
(512, 37000)Used by 3
Constructors
1dunder
__init__
→None__init__(vocab_size: int = 37000, hidden_size: int = 512, num_hidden_layers: int = 6, num_attention_heads: int = 8, intermediate_size: int = 2048, hidden_act: TextActivation = 'relu', max_position_embeddings: int = 5000, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, label_smoothing: float = 0.1, decoder_vocab_size: int | None = None, num_decoder_layers: int = 6, share_embeddings: bool = True, num_labels: int = 2, classifier_dropout: float | None = None, encoder_only: bool = False)