bert_large

→BERTModel

bert_large(pretrained: bool | str = False, weights: BERTLargeWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Large encoder.

Larger of the two original variants from Devlin et al., 2018 Table 1: $L=24$ transformer layers, $H=1024$ hidden, $A=16$ attention heads, intermediate width 4096 — roughly 340M parameters. Uses the 30,522-piece WordPiece vocabulary and a maximum sequence length of 512. Achieved SOTA on every GLUE task and SQuAD v1.1/v2.0 at release.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration. Returns a randomly initialised model today.

weights(BERTLargeWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides forwarded into the underlying config.

Returns

BERTModel

Encoder trunk configured with the BERT-Large size and any overrides.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805), Table 1 (BERT\ LARGE).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_large
>>> model = bert_large().eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])   # [CLS] hello world [SEP]
>>> out = model(input_ids)
>>> out.last_hidden_state.shape, out.pooler_output.shape
((1, 4, 1024), (1, 1024))

Used by 1

lucid.models.text.bert

>>> import lucid >>> from lucid.models.text.bert import bert_large >>> model = bert_large().eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) # [CLS] hello world [SEP] >>> out = model(input_ids) >>> out.last_hidden_state.shape, out.pooler_output.shape ((1, 4, 1024), (1, 1024))