class
CharTokenizerFast
extends
TokenizerCharTokenizerFast(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)C++-backed character tokenizer; bit-identical to CharTokenizer.
Same vocab format + semantics as CharTokenizer; the
per-codepoint encode loop runs in C++ via
lucid._C.engine.utils.tokenizer.CharTokenizer.
Parameters
vocabdict[str, int]= NonePre-built codepoint → id map. If omitted the tokenizer is
constructed empty (call
train before encoding).Special-token registry; configure
unk for OOV fallback.Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed character tokenizer.
Parameters
vocabdict[str, int] or None= NonePre-built codepoint → id map.
None or empty
constructs the C++ backend with an empty vocab; call
train to populate.Special-token registry.
Notes
Caches the Python-side vocab so get_vocab and
id_to_token avoid a binding round-trip; both halves
are kept in sync by train.
Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing vocab.txt.
Parameters
directorystrDirectory previously written by
save.Override for the on-disk special tokens.
Returns
CharTokenizerFastNew instance configured with the loaded vocab + specials.
Instance methods
4Return a copy of the codepoint → id map.
Return the codepoint string for token_id or None.
Persist as vocab.txt + unified tokenizer.json.
Train in C++; mirror the resulting vocab back into Python state.
Parameters
corpusiterable of strMaterialised into a list before hand-off to C++.
vocab_sizeint= 1000Maximum number of distinct codepoints to retain.