WordTokenizer
TokenizerWordTokenizer(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) word-level tokenizer with UNK fallback.
Splits the input on ASCII whitespace runs, looks each word up
in the vocab, and either emits the configured unk token or
raises ValueError on out-of-vocab inputs.
For production / latency-sensitive use, prefer the matching
WordTokenizerFast — same algorithm, same vocab format,
same OOV-error semantics, but the hot loop runs in C++.
Parameters
vocabdict[str, int]= Nonetrain must be called
before encoding.unk if the caller plans to
encode text containing OOV words. Without it, OOV at
encode time raises ValueError.Notes
train does not pre-seed an UNK token. The caller
is responsible for adding it to both the vocab and the
SpecialTokens registry before encoding OOV text.
Examples
>>> tok = WordTokenizer(special_tokens=SpecialTokens(unk="<unk>"))
>>> tok.train(["the quick brown fox", "<unk>"])
>>> tok.encode("the slow fox") # 'slow' → unk
[0, 4, 3]See Also
WordTokenizerFast—C++-backed sibling with the same surface API.
Used by 1
Constructors
1__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python word tokenizer.
Parameters
vocabdict[str, int] or None= NoneNone or empty leaves the
tokenizer empty (call train to populate).unk for OOV
fallback at encode time.Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory previously written by save.
Parameters
directorystrvocab.txt (mandatory) and
optionally special_tokens_map.json.special_tokens_map.json.Returns
WordTokenizerReconstructed tokenizer.
Instance methods
4Return a copy of the token-string → id map.
Reverse lookup; returns None if token_id is unknown.
Persist vocab + unified tokenizer.json.
Parameters
directorystrvocab.txt, tokenizer.json (via the
base class), and optionally special_tokens_map.json.Collect unique words in insertion order until vocab_size.
Same algorithm as WhitespaceTokenizer.train — pure
insertion-order vocab build with a hard cap. No frequency
cut-off is applied.
Training does not pre-seed an UNK token; the caller is responsible for adding it to the vocab + the special-token registry before encoding OOV text.
Parameters
corpusiterable of strvocab_sizeint= 30 000