bert_base_mlm(pretrained: bool | str = False, weights: BERTBaseMLMWeights | None = None, overrides: object = {})Construct a BERT-Base model with a tied masked-LM head.
Same trunk as bert_base (L=12, H=768, A=12, ~110M parameters),
augmented with the masked-language-modeling prediction head used at
pre-training time. Suitable for continued pre-training on domain
corpora, mask-filling inference, and reproducing the MLM half of the
Devlin et al. (2018) pre-training recipe.
Model Size
Parameters
pretrainedbool= FalseReserved for future weight registration; currently a no-op.
weights(BERTBaseMLMWeights, optional, keyword - only)= NoneExplicit weights enum member; takes precedence over
pretrained.**overridesobject= {}Optional
BERTConfig field overrides forwarded into the
underlying config.Returns
BERTForMaskedLMBERT-Base wrapped with the tied MLM head.
Notes
Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805) §3.1 (task #1).
Examples
>>> import lucid
>>> from lucid.models.text.bert import bert_base_mlm
>>> model = bert_base_mlm().eval()
>>> input_ids = lucid.tensor([[101, 7592, 103, 102]]) # [CLS] hello [MASK] [SEP]
>>> out = model(input_ids)
>>> out.logits.shape # (1, 4, vocab_size=30522)
(1, 4, 30522)