data
RoFormerConfig
extends
LanguageModelConfigRoFormerConfig(vocab_size: int = 50000, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 1536, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, rotary_base: float = 10000.0, position_embedding_type: Literal['rotary'] = 'rotary', type_vocab_size: int = 2, num_labels: int = 2, classifier_dropout: float | None = None)Configuration for every RoFormer variant.
Parameters
rotary_basefloat= 10000.0Frequency base
θ_0 for the rotary embedding
(θ_i = base ** (-2 i / d_head)). 10000.0 per the paper.type_vocab_sizeint= 2Segment-id vocabulary (kept for BERT-parity even
though RoFormer fine-tunes typically feed a single segment).
position_embedding_typeLiteral['rotary']= 'rotary'Always
"rotary" here — kept as a literal
for forward compat with future variants (NTK-aware scaling, etc.).num_labels / classifier_dropoutDownstream classification head knobs.
Used by 3
Constructors
1dunder
__init__
→None__init__(vocab_size: int = 50000, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 1536, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, rotary_base: float = 10000.0, position_embedding_type: Literal['rotary'] = 'rotary', type_vocab_size: int = 2, num_labels: int = 2, classifier_dropout: float | None = None)