class
CharTokenizer
extends
TokenizerCharTokenizer(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) character-level tokenizer.
Parameters
vocabdict[str, int]= NonePre-built vocab; if omitted, an empty tokenizer is
constructed (call
train before encoding).Special-token registry. Configuring
unk enables OOV
fallback (otherwise unknown codepoints are dropped).Examples
>>> from lucid.utils.tokenizer import CharTokenizer
>>> tok = CharTokenizer()
>>> tok.train(["the quick brown fox", "the lazy dog sleeps"],
... vocab_size=64)
>>> tok.encode("the quick dog")[:5]
[0, 1, 2, 3, 4]
>>> tok.decode(tok.encode("the quick dog"))
'the quick dog'Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python character tokenizer.
Parameters
vocabdict[str, int] or None= NonePre-built codepoint → id map;
None or empty means
empty tokenizer (call train to populate).Special-token registry; configure
unk for OOV
fallback during encode.Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing vocab.txt.
Parameters
directorystrDirectory previously written by
save.Override for the on-disk special tokens.
Returns
CharTokenizerNew instance configured with the loaded vocab + specials.
Instance methods
4Return a copy of the codepoint → id map.
Return the codepoint string for token_id or None.
Persist the vocab as vocab.txt + unified tokenizer.json.
Parameters
directorystrOutput directory (created if missing).
Build vocab by collecting unique codepoints in insertion order.
Stops when vocab_size distinct codepoints have been
observed.
Parameters
corpusiterable of strEach item is one document. Generators are consumed once.
vocab_sizeint= 1000Maximum number of distinct codepoints to retain.