Bare BERT encoder returning hidden states and pooled CLS embedding.
Implements the bidirectional transformer encoder of Devlin et al., 2018.
Token, position, and segment embeddings are summed, LayerNormed, and
dropout-regularised, then passed through transformer blocks of
multi-head self-attention plus position-wise feed-forward. A single
tanh-activated linear ("pooler") on the first [CLS] token produces a
sentence-level embedding used by classification heads.
Use this class as the trunk when you want raw hidden states; the
task-specific subclasses (BERTFor*) wrap it with appropriate heads.
Parameters
configBERTConfigBERTConfig for the full field list.Attributes
embeddingsnn.Moduleencodernn.Moduleconfig.num_hidden_layers transformer encoder layers.poolernn.Module[CLS] hidden state.config_classtype[BERTConfig]base_model_prefixstr"bert") under which sub-module checkpoints are nested in
task-head variants — used during weight loading.Notes
Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Self-attention follows the scaled dot-product form
with . Each layer applies multi-head attention, followed by a feed-forward block
each wrapped by a residual connection and post-LayerNorm.
Examples
>>> import lucid
>>> from lucid.models.text.bert import BERTConfig, BERTModel
>>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128, num_attention_heads=2,
... intermediate_size=512)
>>> model = BERTModel(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) # [CLS] hello world [SEP]
>>> out = model(input_ids)
>>> out.last_hidden_state.shape # (B=1, T=4, H=128)
(1, 4, 128)
>>> out.pooler_output.shape # (B=1, H=128)
(1, 128)Used by 2
Constructors
1Instance methods
3forward(input_ids: Tensor | None = None, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, position_ids: Tensor | None = None, inputs_embeds: Tensor | None = None, output_hidden_states: bool = False, head_mask: Tensor | None = None, output_attentions: bool = False)Encode a batch and return the sequence output plus the pooled CLS.
position_ids and inputs_embeds are forwarded to the embedding
layer; see _BERTEmbeddings.forward. output_hidden_states
additionally returns every layer's output, embedding first.
head_mask and output_attentions both need the per-head
attention weights, so requesting either switches that layer to the
unfused softmax(qk^T) v path — the fused kernel never
materialises the (B, H, T, T) matrix, which is exactly why it
is fast. Neither flag changes the result when off, and the fused
path stays the default.