data

BERTConfig

extendsLanguageModelConfig

BERTConfig(vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 512, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, type_vocab_size: int = 2, position_embedding_type: Literal['absolute'] = 'absolute', num_labels: int = 2, classifier_dropout: float | None = None)

source edit

Configuration dataclass for every BERT variant.

Frozen dataclass inheriting all common text-model fields from LanguageModelConfig (hidden_act, hidden_dropout, attention_dropout, layer_norm_eps, initializer_range, tie_word_embeddings, ...) and adding the two knobs unique to the BERT family. Field names and defaults mirror the reference-framework BERTConfig so existing tokenizers and checkpoints map one-to-one onto Lucid weights. Defaults match the original bert-base-uncased release (L=12, H=768, A=12, 30,522 WordPiece vocab) from Devlin et al., 2018.

Parameters

vocab_sizeint= 30522

Size of the WordPiece token vocabulary. Both BERT-Base and BERT-Large use 30,522 cased/uncased WordPieces from Devlin et al. 2018.

hidden_sizeint= 768

Dimensionality

H

of every transformer hidden state. Must be divisible by num_attention_heads.

num_hidden_layersint= 12

Number

L

of transformer encoder layers stacked in the trunk.

num_attention_headsint= 12

Number

A

of parallel attention heads per layer. Per-head dimension is hidden_size // num_attention_heads.

intermediate_sizeint= 3072

Width of the position-wise feed-forward inner projection (typically 4 * hidden_size).

max_position_embeddingsint= 512

Maximum sequence length supported by the absolute position-embedding table. Inputs longer than this raise a shape error at lookup.

pad_token_idint or None= 0

Token id treated as padding. When set, the Embedding layer zeroes out that row (no gradient).

type_vocab_sizeint= 2

Size of the segment-id vocabulary used by the next-sentence objective. 2 corresponds to the canonical BERT (segment A / B) setup; single-segment fine-tunes still use the full table but only ever feed zeros.

position_embedding_typeabsolute= "absolute"

Position encoding flavour. "absolute" is the only one shipped today — relative-position variants (T5-style, RoFormer-style) live in their own families.

num_labelsint= 2

Number of output classes for downstream classification heads (BERTForSequenceClassification / BERTForTokenClassification).

classifier_dropoutfloat or None= None

Optional dropout probability inserted before the task-specific classifier. When None, the head falls back to hidden_dropout.

Attributes

model_typestr

Class-level identifier "bert" used by the model registry to look up the matching factory at load time.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).

Validation runs in __post_init__ and raises ValueError for type_vocab_size <= 0, num_labels <= 0, or classifier_dropout outside $[0, 1)$ .

Examples

>>> from lucid.models.text.bert import BERTConfig
>>> cfg = BERTConfig()          # BERT-Base defaults
>>> cfg.hidden_size, cfg.num_hidden_layers, cfg.num_attention_heads
(768, 12, 12)
>>> large = BERTConfig(
...     hidden_size=1024,
...     num_hidden_layers=24,
...     num_attention_heads=16,
...     intermediate_size=4096,
... )
>>> large.vocab_size
30522

Used by 3

Constructors

dunder

init

→None

__init__(vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 512, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, type_vocab_size: int = 2, position_embedding_type: Literal['absolute'] = 'absolute', num_labels: int = 2, classifier_dropout: float | None = None)

source edit

BERTConfig(vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 512, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, type_vocab_size: int = 2, position_embedding_type: Literal['absolute'] = 'absolute', num_labels: int = 2, classifier_dropout: float | None = None)

>>> from lucid.models.text.bert import BERTConfig >>> cfg = BERTConfig() # BERT-Base defaults >>> cfg.hidden_size, cfg.num_hidden_layers, cfg.num_attention_heads (768, 12, 12) >>> large = BERTConfig( ... hidden_size=1024, ... num_hidden_layers=24, ... num_attention_heads=16, ... intermediate_size=4096, ... ) >>> large.vocab_size 30522

__init__(vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: TextActivation = 'gelu', max_position_embeddings: int = 512, pad_token_id: int | None = 0, bos_token_id: int | None = None, eos_token_id: int | None = None, hidden_dropout: float = 0.1, attention_dropout: float = 0.1, initializer_range: float = 0.02, layer_norm_eps: float = 1e-12, tie_word_embeddings: bool = True, use_cache: bool = True, type_vocab_size: int = 2, position_embedding_type: Literal['absolute'] = 'absolute', num_labels: int = 2, classifier_dropout: float | None = None)