class

BPETokenizer

extends_BPECommonMixinTokenizer

BPETokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Reference (pure-Python) BPE tokenizer.

Implements the classical Sennrich-2016 BPE encode loop directly in Python. Easy to read, easy to step through with a debugger, works on every platform (no C++ build needed).

For production / latency-sensitive use, prefer the matching BPETokenizerFast — same algorithm, same vocab format, bit-identical encode outputs, but the hot loop is in C++.

Parameters

vocabdict[str, int]

Token-string → id map. Must include every single character that any merge could produce (so the algorithm has a valid starting symbol sequence).

mergeslist[tuple[str, str]]

Ordered BPE merge pairs — index = rank (lower = higher priority, applied first during encoding).

normalizerNormalizer= None

Pre-encode text normalisation chain. Default lucid.utils.tokenizer._normalizers.NFC.

pre_tokenizerPreTokenizer= None

Chunk-splitter that turns the normalised text into the sequence of strings the BPE algorithm processes one at a time. Default lucid.utils.tokenizer._pre_tokenizers.WhitespaceSplit.

special_tokensSpecialTokens= None

Special-token registry — see lucid.utils.tokenizer.SpecialTokens.

Used by 2

Constructors

dunder

init

→None

__init__(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Construct a pure-Python BPE tokenizer.

Parameters

vocabdict[str, int]

Token-string → id map; must cover every starting symbol referenced by merges.

mergeslist of (str, str)

Ordered merge pairs; index = rank (lower wins).

normalizer(Normalizer or None, optional, keyword - only)= None

Pre-encode normalisation chain. Defaults to NFC.

pre_tokenizer(PreTokenizer or None, optional, keyword - only)= None

Chunk splitter. Defaults to WhitespaceSplit.

special_tokens(SpecialTokens or None, optional, keyword - only)= None

Special-token registry.

Notes

Mutates internal state: copies vocab / merges, builds the _pair_to_merge compiled lookup, and registers special tokens via the base Tokenizer constructor.

Properties

prop

algo

→str

algo: str

source edit

Algorithm identifier (always "bpe").

Returns

str

Constant string "bpe" — used by the unified tokenizer.json writer to label the algorithm.

prop

vocab_size

→int

vocab_size: int

source edit

Number of tokens currently registered in the vocab.

Returns

int

len(self._vocab) — includes special tokens that were registered into the main vocab.

Class methods

cls

from_file

→BPETokenizer

from_file(directory: str, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Load from a directory containing either tokenizer.json or the legacy vocab.json + merges.txt pair.

Unified format wins when present.

Parameters

directorystr

Path containing the on-disk BPE checkpoint — either the unified HF tokenizer.json or the legacy pair (vocab.json + merges.txt). Optionally also special_tokens_map.json for the special-token registry.

normalizer(Normalizer, optional, keyword - only)= None

Override the encode-time normalisation chain. Defaults to lucid.utils.tokenizer._normalizers.NFC when not supplied.

pre_tokenizer(PreTokenizer, optional, keyword - only)= None

Override the chunk splitter applied after normalisation. Defaults to lucid.utils.tokenizer._pre_tokenizers.WhitespaceSplit.

special_tokens(SpecialTokens, optional, keyword - only)= None

Override the special-token registry. When None (the default), parsed from special_tokens_map.json if present.

Returns

BPETokenizer

Freshly-constructed instance ready for encode / decode.

Instance methods

get_vocab

→dict[str, int]

get_vocab()

source edit

Return a shallow copy of the token → id map.

Returns

dict[str, int]

Copy of the internal vocab; mutating it does not affect the tokenizer.

id_to_token

→str or None

id_to_token(token_id: int)

source edit

Look up the surface string for a token id.

Parameters

token_idint

Vocab id.

Returns

str or None

The token string for token_id, or None if the id is not in the reverse table.

save

→None

save(directory: str)

source edit

Persist as both legacy (vocab.json + merges.txt) AND unified (tokenizer.json) formats.

Writing both means the resulting directory loads back via from_pretrained regardless of which format the caller looks for first — and matches how HF distributes BPE checkpoints on the Hub.

train

→None

train(corpus: Iterable[str], vocab_size: int = 30000)

source edit

Re-train this tokenizer from scratch on corpus.

Implements the classical Sennrich-2016 training loop:

Pre-tokenize each document (using the configured pre-tokenizer chain) into a frequency-counted set of words.
Seed the vocab with every distinct codepoint appearing in any word.
Iteratively count adjacent symbol-pair frequencies + merge the most frequent pair, growing the vocab by 1 each step until vocab_size is reached or no pair appears more than once.

Replaces _vocab + _merges in place; previous contents are discarded.

Parameters

corpusiterable of str

Each item is one document (or chunk thereof). Generators are consumed exactly once.

vocab_sizeint= 30 000

Target total vocab size (chars + merges). Stops early if no more merges occur.

class

BPETokenizer

extends_BPECommonMixinTokenizer

BPETokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Reference (pure-Python) BPE tokenizer.

Implements the classical Sennrich-2016 BPE encode loop directly in Python. Easy to read, easy to step through with a debugger, works on every platform (no C++ build needed).

For production / latency-sensitive use, prefer the matching BPETokenizerFast — same algorithm, same vocab format, bit-identical encode outputs, but the hot loop is in C++.

Parameters

vocabdict[str, int]

Token-string → id map. Must include every single character that any merge could produce (so the algorithm has a valid starting symbol sequence).

mergeslist[tuple[str, str]]

Ordered BPE merge pairs — index = rank (lower = higher priority, applied first during encoding).

normalizerNormalizer= None

Pre-encode text normalisation chain. Default lucid.utils.tokenizer._normalizers.NFC.

pre_tokenizerPreTokenizer= None

Chunk-splitter that turns the normalised text into the sequence of strings the BPE algorithm processes one at a time. Default lucid.utils.tokenizer._pre_tokenizers.WhitespaceSplit.

special_tokensSpecialTokens= None

Special-token registry — see lucid.utils.tokenizer.SpecialTokens.

Used by 2

Constructors

dunder

init

→None

__init__(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Construct a pure-Python BPE tokenizer.

Parameters

vocabdict[str, int]

Token-string → id map; must cover every starting symbol referenced by merges.

mergeslist of (str, str)

Ordered merge pairs; index = rank (lower wins).

normalizer(Normalizer or None, optional, keyword - only)= None

Pre-encode normalisation chain. Defaults to NFC.

pre_tokenizer(PreTokenizer or None, optional, keyword - only)= None

Chunk splitter. Defaults to WhitespaceSplit.

special_tokens(SpecialTokens or None, optional, keyword - only)= None

Special-token registry.

Notes

Mutates internal state: copies vocab / merges, builds the _pair_to_merge compiled lookup, and registers special tokens via the base Tokenizer constructor.

Properties

prop

algo

→str

algo: str

source edit

Algorithm identifier (always "bpe").

Returns

str

Constant string "bpe" — used by the unified tokenizer.json writer to label the algorithm.

prop

vocab_size

→int

vocab_size: int

source edit

Number of tokens currently registered in the vocab.

Returns

int

len(self._vocab) — includes special tokens that were registered into the main vocab.

Class methods

cls

from_file

→BPETokenizer

from_file(directory: str, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Load from a directory containing either tokenizer.json or the legacy vocab.json + merges.txt pair.

Unified format wins when present.

Parameters

directorystr

normalizer(Normalizer, optional, keyword - only)= None

Override the encode-time normalisation chain. Defaults to lucid.utils.tokenizer._normalizers.NFC when not supplied.

pre_tokenizer(PreTokenizer, optional, keyword - only)= None

Override the chunk splitter applied after normalisation. Defaults to lucid.utils.tokenizer._pre_tokenizers.WhitespaceSplit.

special_tokens(SpecialTokens, optional, keyword - only)= None

Override the special-token registry. When None (the default), parsed from special_tokens_map.json if present.

Returns

BPETokenizer

Freshly-constructed instance ready for encode / decode.

Instance methods

get_vocab

→dict[str, int]

get_vocab()

source edit

Return a shallow copy of the token → id map.

Returns

dict[str, int]

Copy of the internal vocab; mutating it does not affect the tokenizer.

id_to_token

→str or None

id_to_token(token_id: int)

source edit

Look up the surface string for a token id.

Parameters

token_idint

Vocab id.

Returns

str or None

The token string for token_id, or None if the id is not in the reverse table.

save

→None

save(directory: str)

source edit

Persist as both legacy (vocab.json + merges.txt) AND unified (tokenizer.json) formats.

Writing both means the resulting directory loads back via from_pretrained regardless of which format the caller looks for first — and matches how HF distributes BPE checkpoints on the Hub.

train

→None

train(corpus: Iterable[str], vocab_size: int = 30000)

source edit

Re-train this tokenizer from scratch on corpus.

Implements the classical Sennrich-2016 training loop:

Pre-tokenize each document (using the configured pre-tokenizer chain) into a frequency-counted set of words.
Seed the vocab with every distinct codepoint appearing in any word.
Iteratively count adjacent symbol-pair frequencies + merge the most frequent pair, growing the vocab by 1 each step until vocab_size is reached or no pair appears more than once.

Replaces _vocab + _merges in place; previous contents are discarded.

Parameters

corpusiterable of str

Each item is one document (or chunk thereof). Generators are consumed exactly once.

vocab_sizeint= 30 000

Target total vocab size (chars + merges). Stops early if no more merges occur.