class

BERTForCausalLM

extendsPretrainedModel

BERTForCausalLM(config: BERTConfig)

source edit

BERT trunk repurposed as a left-to-right (causal) language model.

Standard BERT attends bidirectionally; this wrapper injects a lower-triangular causal mask on top of the existing additive attention/padding mask so the same encoder weights behave as a decoder. The LM head is the same tied projection used by BERTForMaskedLM. Use this class when you want to apply pre-trained BERT weights to a generative or sequence-continuation task.

Parameters

configBERTConfig

BERT hyperparameters. config.tie_word_embeddings (default True) ties the LM decoder weight to the input embedding matrix.

Attributes

bertBERTModel

Underlying transformer trunk; only embeddings and encoder are invoked in forward (the pooler is bypassed).

clsnn.Module

Tied LM prediction head — same architecture as BERTForMaskedLM.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805); causal adaptation follows the standard left-to-right LM masking scheme.

The additive causal mask $M \in \mathbb{R}^{T \times T}$ satisfies

M_{ij} = \begin{cases} 0, & j \le i \\ -10^{4}, & j > i \end{cases}

and is broadcast against a padding mask when present. Loss (when labels is supplied) uses the standard next-token shift:

\mathcal{L}_{\mathrm{CLM}} = -\frac{1}{B(T-1)} \sum_{b,t} \log p_{\theta}(y_{b, t+1} \mid x_{b,\le t}),

with positions labelled -100 excluded.

Examples

>>> import lucid
>>> from lucid.models.text.bert import BERTConfig, BERTForCausalLM
>>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = BERTForCausalLM(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]])
>>> out = model(input_ids)
>>> out.logits.shape   # (B=1, T=4, V=30522)
(1, 4, 30522)

Used by 1

lucid.models.text.bert

Constructors

dunder

init

→None

__init__(config: BERTConfig)

source edit

Instance methods

forward

→CausalLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.bert import BERTConfig, BERTForCausalLM >>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = BERTForCausalLM(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) >>> out = model(input_ids) >>> out.logits.shape # (B=1, T=4, V=30522) (1, 4, 30522)