tokenizer
20 memberslucid.utils.tokenizer`lucid.utils.tokenizer` — text tokenization sub-package.
Every tokenizer comes in two flavours:
XxxTokenizer— pure-Python reference implementation. Easy to read, easy to debug, easy to extend with custom normalizers.XxxTokenizerFast— C++-backed vialucid._C.engine.utils.tokenizer. Same vocab format, same encode output bit-for-bit, but the algorithm hot loop runs in C++.
Both flavours share the Hugging Face-compatible on-disk format
(vocab.json + merges.txt legacy or unified
tokenizer.json) so any published HF checkpoint loads without
modification. The Tokenizer base class layers a uniform
HF-style API on top — Tokenizer.encode /
Tokenizer.decode / batch versions / HF-style
Tokenizer.__call__ with padding + truncation +
return_tensors='lucid'.
Notes
========== =============== ============================= ==========
algo modules used by status
========== =============== ============================= ==========
BPE _bpe (raw BPE — base for byte-BPE) ✅ landed
ByteLevel _byte_bpe GPT, GPT-2, RoBERTa, BART ⏳ planned
WordPiece _wordpiece BERT, RoFormer, DistilBERT ⏳ planned
Unigram _unigram T5, LLaMA, Mistral, mBART ⏳ planned
========== =============== ============================= ==========
Per-model wrappers live in each lucid.models.text.<family>
package's _tokenizer/ directory and subclass the algorithm
matching that family (e.g. BERTTokenizer ←
WordPieceTokenizer).
Classes
SpecialTokens2 methodsCanonical special-token registry.
Tokenizer22 methodsAbstract base for every tokenizer in lucid.utils.tokenizer.
BPETokenizer8 methodsReference (pure-Python) BPE tokenizer.
BPETokenizerFast8 methodsC++-backed BPE tokenizer.
ByteTokenizer8 methodsReference (pure-Python) byte-level tokenizer.
ByteTokenizerFast8 methodsC++-backed byte tokenizer; bit-identical to ByteTokenizer.
ByteLevelBPETokenizer4 methodsPure-Python byte-level BPE (GPT-2 / RoBERTa convention).
ByteLevelBPETokenizerFast4 methodsC++-backed byte-level BPE tokenizer.
CharTokenizer8 methodsReference (pure-Python) character-level tokenizer.
CharTokenizerFast8 methodsC++-backed character tokenizer; bit-identical to CharTokenizer.
RegexTokenizer9 methodsReference (pure-Python) regex tokenizer.
RegexTokenizerFast9 methodsC++-backed regex tokenizer.
UnigramTokenizer9 methodsReference (pure-Python) Unigram tokenizer.
UnigramTokenizerFast9 methodsC++-backed Unigram tokenizer.
WhitespaceTokenizer8 methodsReference (pure-Python) whitespace-split tokenizer.
WhitespaceTokenizerFast8 methodsC++-backed whitespace tokenizer; bit-identical to
WhitespaceTokenizer.
WordTokenizer8 methodsReference (pure-Python) word-level tokenizer with UNK fallback.
WordTokenizerFast8 methodsC++-backed word tokenizer with UNK fallback.
WordPieceTokenizer8 methodsReference (pure-Python) WordPiece tokenizer.
WordPieceTokenizerFast8 methodsC++-backed WordPiece tokenizer.