BPETokenizerFast
_BPECommonMixinTokenizerBPETokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)C++-backed BPE tokenizer.
Identical algorithm + vocab format to BPETokenizer; the
hot loop (per-chunk merge application) runs in C++ via the engine
BPE binding. Encode outputs are bit-identical for the same
vocab + same merges + same normalizer + same pre-tokenizer.
Use this in production training / inference. Use
BPETokenizer for debugging / extending the algorithm
with custom Python-only normalizers without touching C++.
Parameters
SametransparentlyUsed by 2
Constructors
1__init__
→None__init__(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed BPE tokenizer.
Parameters
vocabdict[str, int]mergeslist of (str, str)normalizer(Normalizer or None, optional, keyword - only)= NoneNFC.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneWhitespaceSplit._sync_special_tokens_to_cpp.Notes
Builds the C++ BPE backend once and caches it on _cpp;
the Python-side _id_to_token is kept in sync so decode
avoids a binding round-trip.
Properties
2Algorithm identifier (always "bpe").
Returns
strConstant string "bpe".
Number of tokens in the C++ side vocab.
Returns
intResult of self._cpp.vocab_size() — authoritative
count for the live tokenizer (kept in sync with the
Python cache after every train).
Class methods
1from_file(directory: str, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Identical loader to BPETokenizer.from_file; the
only difference is the returned class (and hence the encode
backend — C++ instead of pure-Python merge loop).
Parameters
directorystrtokenizer.json or the legacy pair
(vocab.json + merges.txt). Optionally also
special_tokens_map.json for the special-token registry.normalizer(Normalizer, optional, keyword - only)= Nonelucid.utils.tokenizer._normalizers.NFC.pre_tokenizer(PreTokenizer, optional, keyword - only)= Nonelucid.utils.tokenizer._pre_tokenizers.WhitespaceSplit.None (the
default), parsed from special_tokens_map.json if
present in directory.Returns
BPETokenizerFastFreshly-constructed C++-backed instance ready for encode / decode.
Instance methods
4Return a shallow copy of the token → id map.
Returns
dict[str, int]Copy of the Python-side cache, kept in sync with the C++
side by __init__ and train.
Notes
Reads from the Python cache to avoid a binding round-trip; both halves are updated atomically.
Look up the surface string for a token id.
Parameters
token_idintReturns
str or NoneThe token string for token_id, or None if the id
is not in the reverse table.
Same format as BPETokenizer.save.
Re-train in C++.
Materialises corpus into a list before handing off (the
C++ binding takes std::vector<std::string>); for very
large corpora the user is responsible for chunking. After
training, the Python-side vocab / merges caches are refreshed
from the C++ side so subsequent encodes see the new state.
Parameters
corpusiterable of strvocab_size(int, optional, keyword - only)= 30000