RoFormerConfig
LanguageModelConfigRoFormerConfig(vocab_size: int = 50000, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 1536, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, embedding_size: int | None = None, rotary_base: float = 10000.0, rotary_value: bool = False, position_embedding_type: Literal['rotary'] = 'rotary', type_vocab_size: int = 2, num_labels: int = 2, classifier_dropout: float | None = None)Configuration for every RoFormer variant.
Args:
embedding_size: Width of the word and token-type tables when the
checkpoint factorises the embedding. None — the default —
ties them to hidden_size; when the two differ the tables
and their LayerNorm are sized here and an
embeddings_project linear maps up to hidden_size.
rotary_base: Frequency base θ_0 for the rotary embedding
(θ_i = base ** (-2 i / d_head)). 10000.0 per the paper.
rotary_value: Also rotate V, not only Q and K. Off by
default because the paper's Eq. 16 rotates the two, and every
released checkpoint was trained that way; the switch exists
because the reference implementation carries it.
type_vocab_size: Segment-id vocabulary (kept for BERT-parity even
though RoFormer fine-tunes typically feed a single segment).
position_embedding_type: Always "rotary" here — kept as a literal
for forward compat with future variants (NTK-aware scaling, etc.).
num_labels: Number of classes the sequence-classification head
projects to.
classifier_dropout: Dropout applied before that head.
Examples
>>> from lucid.models.text.roformer._config import RoFormerConfig
>>> cfg = RoFormerConfig()
>>> cfg.model_type
'roformer'
>>> cfg.hidden_size, cfg.vocab_size
(768, 50000)Used by 3
Constructors
1__init__
→None__init__(vocab_size: int = 50000, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 1536, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, embedding_size: int | None = None, rotary_base: float = 10000.0, rotary_value: bool = False, position_embedding_type: Literal['rotary'] = 'rotary', type_vocab_size: int = 2, num_labels: int = 2, classifier_dropout: float | None = None)