data
SpecialTokens
SpecialTokens(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())Canonical special-token registry.
Each slot is either None (the algorithm doesn't define that
token — e.g. GPT-2 has no pad by default) or the token's
string surface form. The extra map holds any non-canonical
specials (e.g. <|endoftext|> for GPT-2, <|im_start|>
for ChatML) keyed by name.
String values are surface forms (what appears in text); the corresponding ids are looked up in the tokenizer's vocab at construction time.
Used by 15
- lucid.models.text.bert._tokenizer
- lucid.models.text.gpt._tokenizer
- lucid.models.text.gpt2._tokenizer
- lucid.models.text.roformer._tokenizer
- lucid.utils.tokenizer
- lucid.utils.tokenizer._bpe
- lucid.utils.tokenizer._byte
- lucid.utils.tokenizer._byte_bpe
- lucid.utils.tokenizer._char
- lucid.utils.tokenizer._lookup_common
- lucid.utils.tokenizer._regex
- lucid.utils.tokenizer._unigram
… 3 more
Constructors
1dunder
__init__
→None__init__(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())Initialise the special-token registry; subclasses must populate the vocab + algorithm state before calling this.