WordPieceTokenizerFast
_WordPieceCommonMixinTokenizerWordPieceTokenizerFast(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)C++-backed WordPiece tokenizer.
Identical algorithm + vocab format to WordPieceTokenizer;
the hot loops (greedy longest-match encode + training) run in C++
via the engine WordPiece binding. Encode outputs are
bit-identical for the same vocab + same normalizer + same
pre-tokenizer.
The Python side still handles normalisation + pre-tokenization (so user-defined Python normalizers compose with the C++ backend) and the BERT-style decode join.
Parameters
SameconstructedSee Also
WordPieceTokenizer—Pure-Python reference; same vocab format.
Used by 3
Constructors
1__init__
→None__init__(vocab: dict[str, int], unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed WordPiece tokenizer.
Parameters
vocabdict[str, int]unk_tokenstr= "[UNK]"WordPieceTokenizer.continuing_prefixstr= "##"WordPieceTokenizer.max_chars_per_wordint= 100WordPieceTokenizer.normalizer(Normalizer or None, optional, keyword - only)= NoneBERTNormalizer.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneWhitespacePunctuationSplit.SpecialTokens(unk=unk_token).Notes
Constructs the C++ WordPiece backend once and caches it
on _cpp; the Python-side _id_to_token mirror enables fast
decode without round-trips.
Properties
2Class methods
1from_file(directory: str, unk_token: str = '[UNK]', continuing_prefix: str = '##', max_chars_per_word: int = 100, normalizer: Normalizer | None = None, pre_tokenizer: PreTokenizer | None = None, special_tokens: SpecialTokens | None = None)Identical loader to WordPieceTokenizer.from_file;
the only difference is the returned class (and hence the
encode backend).
Parameters
directorystrvocab.txt.unk_token(str, optional, keyword - only)= "[UNK]"WordPieceTokenizer.from_file.continuing_prefix(str, optional, keyword - only)= "##"WordPieceTokenizer.from_file.max_chars_per_word(int, optional, keyword - only)= 100WordPieceTokenizer.from_file.normalizer(Normalizer or None, optional, keyword - only)= NoneWordPieceTokenizer.from_file.pre_tokenizer(PreTokenizer or None, optional, keyword - only)= NoneWordPieceTokenizer.from_file.WordPieceTokenizer.from_file.Returns
WordPieceTokenizerFastA new C++-backed tokenizer populated from disk.
Raises
FileNotFoundErrorvocab.txt is missing.Instance methods
4Return a copy of the token → id map from the Python cache.
Returns
dict[str, int]Shallow copy; kept in sync with the C++ backend by
__init__ and train.
Look up the surface string for a token id.
Parameters
token_idintReturns
str or NoneThe token string, or None if unknown.
Same format as WordPieceTokenizer.save.
Re-train in C++ via WordPiece.train.
Pre-encodes the corpus through the configured normalizer + pre-tokenizer in Python so the C++ training loop sees already-normalized, whitespace-separated words (its internal splitter is also whitespace, so this composes correctly). After training, the Python-side vocab cache is refreshed from the C++ side so subsequent encodes see the new state.
Parameters
corpusiterable of strvocab_sizeint= 30 000