WordTokenizerFast
TokenizerWordTokenizerFast(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)C++-backed word tokenizer with UNK fallback.
Identical surface API and OOV-error semantics to
WordTokenizer; the whitespace scan + dict lookup run
in C++ via lucid._C.engine.utils.tokenizer.WordTokenizer.
Parameters
vocabdict[str, int]= Nonetrain must be
called before encoding.unk to allow OOV input; the id
is mirrored into the C++ registry so the C++ OOV fall-back
path agrees with the Python-side OOV check.Notes
The C++ LookupTokenizer.encode silently substitutes UNK for
OOV words when one is configured, or drops them otherwise. To
match the Python reference's explicit error for unconfigured
UNK, _encode_one performs an OOV pre-check before
delegating.
See Also
WordTokenizer—Pure-Python reference sibling.
Used by 1
Constructors
1__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed word tokenizer.
Parameters
vocabdict[str, int] or None= NoneNone or empty constructs
the C++ backend with an empty vocab._sync_specials_to_cpp so the C++ encode
fall-back path sees the same UNK id.Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Identical loader to WordTokenizer.from_file.
The only difference is the returned class (and hence the encode backend).
Parameters
directorystrvocab.txt.special_tokens_map.json.Returns
WordTokenizerFastReconstructed tokenizer.
Instance methods
4Return a copy of the cached token-string → id map.
Reverse lookup; returns None if token_id is unknown.
Same on-disk format as WordTokenizer.save.
Parameters
directorystrRe-train in C++; refresh Python-side caches afterwards.
Parameters
corpusiterable of strlist before being handed to the
C++ binding.vocab_sizeint= 30 000