transformer_base_token_cls

→TransformerForTokenClassification

transformer_base_token_cls(pretrained: bool = False, overrides: object = {})

source edit

Construct a Transformer "base" with a per-token classification head.

Same trunk as transformer_base (N=6, d_model=512, h=8); only the encoder half is used in forward. Every source-side token's hidden state is dropout-regularised and projected through a linear of output width config.num_labels, with the standard masked cross-entropy inherited from MaskedLMMixin. Use for sequence-labelling tasks (NER, POS) when an encoder-only Vaswani trunk is preferred to BERT.

Model Size

Parameters

pretrainedbool= False

Reserved for future weight registration; currently a no-op.

**overridesobject= {}

Optional TransformerConfig field overrides forwarded into the underlying config. Pass num_labels=N to set the tag set size.

Returns

TransformerForTokenClassification

Encoder-only Transformer trunk wrapped with a per-token classifier head.

Notes

Reference: Vaswani et al., "Attention Is All You Need", NeurIPS, 2017.

Loss (when labels is provided) is the masked cross-entropy over positions whose label is not -100:

\mathcal{L} = -\frac{1}{|V|} \sum_{(b, t) \in V} \log p_\theta(y_{b, t} \mid x_b).

Examples

>>> import lucid
>>> from lucid.models.text.transformer import transformer_base_token_cls
>>> model = transformer_base_token_cls(num_labels=9).eval()
>>> input_ids = lucid.tensor([[1, 234, 567, 2]])
>>> out = model(input_ids)
>>> out.logits.shape   # (1, T=4, num_labels=9)
(1, 4, 9)

Used by 1

lucid.models.text.transformer

>>> import lucid >>> from lucid.models.text.transformer import transformer_base_token_cls >>> model = transformer_base_token_cls(num_labels=9).eval() >>> input_ids = lucid.tensor([[1, 234, 567, 2]]) >>> out = model(input_ids) >>> out.logits.shape # (1, T=4, num_labels=9) (1, 4, 9)