RegexTokenizerFast
TokenizerRegexTokenizerFast(pattern: str, vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)C++-backed regex tokenizer.
Identical surface API to RegexTokenizer; the per-chunk
regex scan runs in C++ via
lucid._C.engine.utils.tokenizer.RegexTokenizer. Use
this in production training / inference where encode throughput
matters.
Parameters
patternstrstd::regex
default), not the Python re dialect — see Notes.vocabdict[str, int]= Nonelucid.utils.tokenizer.SpecialTokens. The unk
id is mirrored into the C++ side so the OOV fall-back path
is consistent across both flavours.Notes
The Fast flavour uses std::regex (ECMAScript dialect) while
the Python flavour uses re — patterns that lean on
Python-specific features (named-group syntax with ?P<name>,
lookbehind variability, etc.) may diverge between the two.
Stick to plain character classes + quantifiers for portable
patterns.
See Also
RegexTokenizer—Pure-Python reference sibling.
Used by 1
Constructors
1__init__
→None__init__(pattern: str, vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed regex tokenizer.
Parameters
patternstrvocabdict[str, int] or None= NoneProperties
3Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Identical loader to RegexTokenizer.from_file.
The only difference is the returned class (and hence the encode backend).
Parameters
directorystrregex_pattern.txt.special_tokens_map.json.Returns
RegexTokenizerFastReconstructed tokenizer.
Instance methods
4Return a copy of the cached token-string → id map.
Reverse lookup; returns None if token_id is unknown.
Same on-disk format as RegexTokenizer.save.
Parameters
directorystrRe-train in C++; refresh Python-side caches afterwards.
Parameters
corpusiterable of strlist before being handed to the
C++ binding.vocab_sizeint= 30 000