bert_large_mlm

→BERTForMaskedLM

bert_large_mlm(pretrained: bool | str = False, weights: BERTLargeMLMWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Large model with a tied masked-LM head.

Same trunk as bert_large (L=24, H=1024, A=16, ~340M parameters), augmented with the masked-language-modeling prediction head used at pre-training time.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(BERTLargeMLMWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides forwarded into the underlying config.

Returns

BERTForMaskedLM

BERT-Large wrapped with the tied MLM head.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805) §3.1 (task #1).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_large_mlm
>>> model = bert_large_mlm().eval()
>>> input_ids = lucid.tensor([[101, 7592, 103, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (1, 4, 30522)
(1, 4, 30522)

Used by 1

lucid.models.text.bert