Abstract base. Each concrete algorithm (BPE / WordPiece / Unigram / ByteLevelBPE) subclasses it + overrides the four pure-virtual hooks. The Python Fast wrapper holds a std::unique_ptr<Tokenizer> and dispatches through the v-table.
Destructor
1Virtual methods
3Special token registry. Default is empty. Subclasses set it during construction or via set_special_tokens.
In-place training from an iterable of text samples. Default throws std::runtime_error (algorithm doesn't support training in C++). BPE / WordPiece overrides should populate the internal vocab + merge table from scratch.
Methods
8Algorithm name — used by save / from_pretrained to route to the right loader. Lower-case, e.g. "bpe", "wordpiece", "unigram", "byte_bpe".
Convert a sequence of token ids back to text. Special token skipping is delegated to the Python wrapper (which knows the skip_special_tokens flag); this raw method outputs every token's surface form including specials.
Batched decode — default loops over decode.
Convert a string of input text into the corresponding token id sequence. Implementations should apply the algorithm-specific pipeline: normalize → pre-tokenize → algorithm-specific encode. Special tokens (BOS/EOS/CLS/SEP) are NOT auto-added — the caller (the Python encode wrapper) decides via add_special_tokens.
Batched encode — default loops over encode. Subclasses can override for parallelism / batched algorithm-specific optimisations.
Vocab introspection: map from token string → id. Used by the Python __call__ for special-token-mask computation and by save for serialisation. Default empty; subclasses build it.
Inverse: id → token string. Default scans get_vocab; subclasses with an O(1) reverse table should override.
Total vocabulary size — number of distinct ids the tokenizer can emit (including special tokens).