class

BERTForNextSentencePrediction

extendsPretrainedModel

BERTForNextSentencePrediction(config: BERTConfig)

source edit

BERT with the standalone next-sentence-prediction head.

Wraps the bidirectional encoder with a single binary linear classifier operating on the pooled [CLS] embedding. This is pre-training task 2 of Devlin et al., 2018 §3.1 in isolation — useful for reproducing historical experiments or as a sanity check for sentence-pair coherence.

NSP was abandoned by RoBERTa, ALBERT, and DeBERTa as offering no downstream value, so prefer BERTForMaskedLM (MLM-only) or BERTForSequenceClassification for new work.

Parameters

configBERTConfig

BERT hyperparameters.

Attributes

bertBERTModel

Underlying bidirectional encoder trunk.

clsnn.Module

NSP head — a single Linear(hidden_size, 2) over the pooled embedding.

Notes

Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805) §3.1 (task #2).

When labels is provided, the loss is the binary cross-entropy

\mathcal{L}_{\mathrm{NSP}} = -\frac{1}{B}\sum_{b=1}^{B} \log p_{\theta}(y_b \mid x_b^{(A)}, x_b^{(B)}),

where $y_b \in \{0, 1\}$ denotes IsNext vs. NotNext.

Examples

>>> import lucid
>>> from lucid.models.text.bert import BERTConfig, BERTForNextSentencePrediction
>>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128,
...                  num_attention_heads=2, intermediate_size=512)
>>> model = BERTForNextSentencePrediction(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 102, 2088, 102]])
>>> token_type_ids = lucid.tensor([[0, 0, 0, 1, 1]])
>>> out = model(input_ids, token_type_ids=token_type_ids)
>>> out.logits.shape   # (B=1, 2)
(1, 2)

Used by 1

lucid.models.text.bert

Constructors

dunder

init

→None

__init__(config: BERTConfig)

source edit

Instance methods

forward

→MaskedLMOutput

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, labels: Tensor | None = None)

source edit

>>> import lucid >>> from lucid.models.text.bert import BERTConfig, BERTForNextSentencePrediction >>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128, ... num_attention_heads=2, intermediate_size=512) >>> model = BERTForNextSentencePrediction(cfg).eval() >>> input_ids = lucid.tensor([[101, 7592, 102, 2088, 102]]) >>> token_type_ids = lucid.tensor([[0, 0, 0, 1, 1]]) >>> out = model(input_ids, token_type_ids=token_type_ids) >>> out.logits.shape # (B=1, 2) (1, 2)