Bare BERT encoder returning hidden states and pooled CLS embedding.
Implements the bidirectional transformer encoder of Devlin et al., 2018.
Token, position, and segment embeddings are summed, LayerNormed, and
dropout-regularised, then passed through transformer blocks of
multi-head self-attention plus position-wise feed-forward. A single
tanh-activated linear ("pooler") on the first [CLS] token produces a
sentence-level embedding used by classification heads.
Use this class as the trunk when you want raw hidden states; the
task-specific subclasses (BERTFor*) wrap it with appropriate heads.
Parameters
configBERTConfigBERTConfig for the full field list.Attributes
embeddingsnn.Moduleencodernn.Moduleconfig.num_hidden_layers transformer encoder layers.poolernn.Module[CLS] hidden state.config_classtype[BERTConfig]base_model_prefixstr"bert") under which sub-module checkpoints are nested in
task-head variants — used during weight loading.Notes
Reference: Devlin, Chang, Lee, and Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", NAACL 2019 (arXiv:1810.04805).
Self-attention follows the scaled dot-product form
with . Each layer applies multi-head attention, followed by a feed-forward block
each wrapped by a residual connection and post-LayerNorm.
Examples
>>> import lucid
>>> from lucid.models.text.bert import BERTConfig, BERTModel
>>> cfg = BERTConfig(num_hidden_layers=2, hidden_size=128, num_attention_heads=2,
... intermediate_size=512)
>>> model = BERTModel(cfg).eval()
>>> input_ids = lucid.tensor([[101, 7592, 2088, 102]]) # [CLS] hello world [SEP]
>>> out = model(input_ids)
>>> out.last_hidden_state.shape # (B=1, T=4, H=128)
(1, 4, 128)
>>> out.pooler_output.shape # (B=1, H=128)
(1, 128)