class
WhitespaceTokenizer
extends
TokenizerWhitespaceTokenizer(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) whitespace-split tokenizer.
Parameters
vocabdict[str, int]= NonePre-built vocab. Empty = empty tokenizer; call
train
first.Special-token registry.
Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python whitespace tokenizer.
Parameters
vocabdict[str, int] or None= NonePre-built vocab.
None or empty = empty tokenizer;
call train to populate.Special-token registry.
Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing vocab.txt.
Parameters
directorystrDirectory previously written by
save.Override for the on-disk special-token map.
Instance methods
4Return a copy of the word → id map.
Return the word for token_id or None if unknown.
Persist as vocab.txt + unified tokenizer.json.
Parameters
directorystrOutput directory (created if missing).
Collect unique whitespace-split words in insertion order.
Parameters
corpusiterable of strEach item is one document. Generators are consumed once.
vocab_sizeint= 30 000Maximum number of distinct words to retain; training stops
as soon as the cap is hit.