bert_base

→BERTModel

bert_base(pretrained: bool | str = False, weights: BERTBaseWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Base encoder.

Canonical mid-size variant from Devlin et al., 2018 Table 1: $L=12$ transformer layers, $H=768$ hidden, $A=12$ attention heads, intermediate width 3072 — roughly 110M parameters. Uses the 30,522-piece WordPiece vocabulary and a maximum sequence length of 512. This is the default starting point for most BERT fine-tuning work.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration. Returns a randomly initialised model today.

weights(BERTBaseWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides (e.g. vocab_size=..., max_position_embeddings=...) forwarded into the underlying config.

Returns

BERTModel

Encoder trunk configured with the BERT-Base size and any overrides.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805), Table 1 (BERT\ BASE).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_base
>>> model = bert_base().eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])   # [CLS] hello world [SEP]
>>> out = model(input_ids)
>>> out.last_hidden_state.shape, out.pooler_output.shape
((1, 4, 768), (1, 768))

Used by 1

lucid.models.text.bert

>>> import lucid >>> from lucid.models.text.bert import bert_base >>> model = bert_base().eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) # [CLS] hello world [SEP] >>> out = model(input_ids) >>> out.last_hidden_state.shape, out.pooler_output.shape ((1, 4, 768), (1, 768))