RegexTokenizer
TokenizerRegexTokenizer(pattern: str, vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) regex tokenizer.
Splits the input by repeatedly applying a compiled regular expression; every match is treated as one chunk and looked up in the vocab. Characters between matches are dropped — there is no canonical reconstruction during decode.
For production / latency-sensitive use, prefer the matching
RegexTokenizerFast — same matching semantics on
portable patterns, but the hot loop runs in C++.
Parameters
patternstrre.finditer. Every match's text becomes a chunk;
unmatched spans are dropped.vocabdict[str, int]= Nonetrain must be called
before encoding.lucid.utils.tokenizer.SpecialTokens. If a
non-None unk is configured, OOV matches map to it;
otherwise OOV matches are silently dropped.Notes
The pattern is compiled once at construction time. Re-assigning the pattern after construction is not supported — build a new instance instead.
Examples
>>> tok = RegexTokenizer(r"\w+")
>>> tok.train(["hello world hello"])
>>> tok.encode("hello world")
[0, 1]See Also
RegexTokenizerFast—C++-backed sibling with the same surface API.
Used by 1
Constructors
1__init__
→None__init__(pattern: str, vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python regex tokenizer.
Parameters
patternstrre.compile.
Each re.findall hit becomes one token candidate.vocabdict[str, int] or None= Noneunk for OOV
handling (OOV matches are dropped otherwise).Properties
3Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory previously written by save.
Parameters
directorystrregex_pattern.txt.
vocab.txt is optional (an empty vocab is used if
absent — call train afterwards).
special_tokens_map.json is consulted if
special_tokens is not provided.special_tokens_map.json.Returns
RegexTokenizerReconstructed tokenizer.
Instance methods
4Return a copy of the token-string → id map.
Reverse lookup; returns None if token_id is unknown.
Persist vocab + regex pattern + unified tokenizer.json.
Parameters
directorystrvocab.txt, regex_pattern.txt,
tokenizer.json (via the base class) and optionally
special_tokens_map.json.Build the vocab in insertion order from corpus matches.
Iterates over corpus, applies the regex to each document,
and assigns each first-seen match a fresh id starting at 0.
Stops as soon as vocab_size distinct tokens have been
collected (no frequency cut-off — the order is purely
insertion order).
Parameters
corpusiterable of strvocab_sizeint= 30 000