data

SpecialTokens

SpecialTokens(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())

source edit

Canonical special-token registry.

Each slot is either None (the algorithm doesn't define that token — e.g. GPT-2 has no pad by default) or the token's string surface form. The extra map holds any non-canonical specials (e.g. <|endoftext|> for GPT-2, <|im_start|> for ChatML) keyed by name.

String values are surface forms (what appears in text); the corresponding ids are looked up in the tokenizer's vocab at construction time.

Examples

Naming a token here is half of it — the other half is that the
token has to be in the vocabulary, and train builds the
vocabulary from the corpus without reserving anything. Seed it:
>>> from lucid.utils.tokenizer import SpecialTokens, WordTokenizer
>>> marks = SpecialTokens(unk="<unk>", pad="<pad>")
>>> tok = WordTokenizer(vocab={"<unk>": 0}, special_tokens=marks)
>>> tok.unk_token_id
0
>>> tok.encode("a word it has never seen")
[0, 0, 0, 0, 0, 0]
Without the seed unk_token_id is None and an out-of-vocabulary
word raises rather than being replaced, which is the safer of the two
since a silent unknown is indistinguishable from a known token.

Used by 16

… 4 more

Constructors

dunder

init

→None

__init__(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())

source edit

Construct the tokenizer. See the class docstring for parameters.

Instance methods

all_tokens

→list[str]

all_tokens()

source edit

Return every special-token surface form (canonical + extra).

data

SpecialTokens

SpecialTokens(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())

source edit

Canonical special-token registry.

String values are surface forms (what appears in text); the corresponding ids are looked up in the tokenizer's vocab at construction time.

Examples

Naming a token here is half of it — the other half is that the
token has to be in the vocabulary, and train builds the
vocabulary from the corpus without reserving anything. Seed it:
>>> from lucid.utils.tokenizer import SpecialTokens, WordTokenizer
>>> marks = SpecialTokens(unk="<unk>", pad="<pad>")
>>> tok = WordTokenizer(vocab={"<unk>": 0}, special_tokens=marks)
>>> tok.unk_token_id
0
>>> tok.encode("a word it has never seen")
[0, 0, 0, 0, 0, 0]
Without the seed unk_token_id is None and an out-of-vocabulary
word raises rather than being replaced, which is the safer of the two
since a silent unknown is indistinguishable from a known token.

Used by 16

… 4 more

Constructors

dunder

init

→None

__init__(pad: str | None = None, unk: str | None = None, bos: str | None = None, eos: str | None = None, mask: str | None = None, sep: str | None = None, extra: dict[str, str] = dict())

source edit

Construct the tokenizer. See the class docstring for parameters.

Instance methods

all_tokens

→list[str]

all_tokens()

source edit

Return every special-token surface form (canonical + extra).