bert_small

→BERTModel

bert_small(pretrained: bool | str = False, weights: BERTSmallWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Small encoder.

Distillation-targeted variant from Turc et al., 2019: $L=4$ layers, $H=512$ hidden, $A=8$ heads, intermediate width 2048. Roughly 29M parameters — a useful mid-point between Mini and Base.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(BERTSmallWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides forwarded into the underlying config.

Returns

BERTModel

Encoder trunk configured with the BERT-Small size and any overrides.

Notes

Reference: Turc, Chang, Lee, and Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_small
>>> model = bert_small().eval()
>>> out = model(lucid.tensor([[101, 7592, 102]]))
>>> out.last_hidden_state.shape   # (1, 3, 512)
(1, 3, 512)

Used by 1

lucid.models.text.bert

>>> import lucid >>> from lucid.models.text.bert import bert_small >>> model = bert_small().eval() >>> out = model(lucid.tensor([[101, 7592, 102]])) >>> out.last_hidden_state.shape # (1, 3, 512) (1, 3, 512)