bert_mini

→BERTModel

bert_mini(pretrained: bool | str = False, weights: BERTMiniWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Mini encoder.

Second-smallest of the Turc et al., 2019 distillation sizes: $L=4$ layers, $H=256$ hidden, $A=4$ heads, intermediate width 1024. Roughly 11M parameters.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

weights(BERTMiniWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides forwarded into the underlying config.

Returns

BERTModel

Encoder trunk configured with the BERT-Mini size and any overrides.

Notes

Reference: Turc, Chang, Lee, and Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_mini
>>> model = bert_mini().eval()
>>> out = model(lucid.tensor([[101, 7592, 102]]))
>>> out.last_hidden_state.shape   # (1, 3, 256)
(1, 3, 256)

Used by 1

lucid.models.text.bert

>>> import lucid >>> from lucid.models.text.bert import bert_mini >>> model = bert_mini().eval() >>> out = model(lucid.tensor([[101, 7592, 102]])) >>> out.last_hidden_state.shape # (1, 3, 256) (1, 3, 256)