bert_tiny

→BERTModel

bert_tiny(pretrained: bool | str = False, weights: BERTTinyWeights | None = None, overrides: object = {})

source edit

Construct a BERT-Tiny encoder.

Smallest of the four distillation-targeted variants from Turc, Chang, Lee, and Toutanova, "Well-Read Students Learn Better", 2019 (arXiv:1908.08962): $L=2$ transformer layers, $H=128$ hidden, $A=2$ heads, intermediate width 512. Roughly 4M parameters — use for CPU/edge inference, latency-bound demos, or unit tests of the BERT pipeline.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration. No checkpoints are wired up yet, so the model is always returned with random initialisation.

weights(BERTTinyWeights, optional, keyword - only)= None

Explicit weights enum member; takes precedence over pretrained.

**overridesobject= {}

Optional BERTConfig field overrides (e.g. vocab_size=..., num_labels=...). Forwarded into a fresh BERTConfig whose defaults match the Tiny size table above.

Returns

BERTModel

Encoder trunk configured with the BERT-Tiny size and any overrides.

Notes

Reference: Turc, Chang, Lee, and Toutanova, "Well-Read Students Learn Better: On the Importance of Pre-training Compact Models", 2019 (arXiv:1908.08962).

Examples

>>> import lucid
>>> from lucid.models.text.bert import bert_tiny
>>> model = bert_tiny().eval()
>>> out = model(lucid.tensor([[101, 7592, 102]]))
>>> out.last_hidden_state.shape   # (1, 3, 128)
(1, 3, 128)

Used by 1

lucid.models.text.bert

>>> import lucid >>> from lucid.models.text.bert import bert_tiny >>> model = bert_tiny().eval() >>> out = model(lucid.tensor([[101, 7592, 102]])) >>> out.last_hidden_state.shape # (1, 3, 128) (1, 3, 128)