class
WhitespaceTokenizerFast
extends
TokenizerWhitespaceTokenizerFast(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)C++-backed whitespace tokenizer; bit-identical to
WhitespaceTokenizer.
Same vocab format + OOV-dropping contract as
WhitespaceTokenizer; the encode loop runs in C++ via
the engine WhitespaceTokenizer binding.
Parameters
vocabdict[str, int]= NonePre-built word → id map. Empty = call
train first.Special-token registry.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed whitespace tokenizer.
Parameters
vocabdict[str, int] or None= NonePre-built word → id map.
None or empty constructs
an empty C++ backend; call train to populate.Special-token registry.
Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing vocab.txt.
Parameters
directorystrDirectory previously written by
save.Override for the on-disk special-token map.
Returns
WhitespaceTokenizerFastA new C++-backed tokenizer populated from disk.
Raises
FileNotFoundErrorIf
vocab.txt is missing.Instance methods
4Return a copy of the word → id map.
Returns
dict[str, int]Shallow copy of the Python-side cache.
Look up the word for token_id.
Parameters
token_idintVocab id.
Returns
str or NoneWord string, or None if unknown.
Persist as vocab.txt + unified tokenizer.json.
Parameters
directorystrOutput directory; created if missing.
Collect unique whitespace-split words via the C++ trainer.
Parameters
corpusiterable of strDocuments to scan. Materialised into a list before
crossing into C++.
vocab_sizeint= 30 000Maximum number of distinct words to retain.
Notes
Mutates _vocab / _id_to_token in place after the C++
side finishes; re-resolves special-token ids against the
new vocab.