class
CharTokenizer
extends
TokenizerCharTokenizer(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Reference (pure-Python) character-level tokenizer.
Parameters
vocabdict[str, int]= NonePre-built vocab; if omitted, an empty tokenizer is
constructed (call
train before encoding).Special-token registry. Configuring
unk enables OOV
fallback (otherwise unknown codepoints are dropped).Used by 1
Constructors
1dunder
__init__
→None__init__(vocab: dict[str, int] | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python character tokenizer.
Parameters
vocabdict[str, int] or None= NonePre-built codepoint → id map;
None or empty means
empty tokenizer (call train to populate).Special-token registry; configure
unk for OOV
fallback during encode.Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing vocab.txt.
Parameters
directorystrDirectory previously written by
save.Override for the on-disk special tokens.
Returns
CharTokenizerNew instance configured with the loaded vocab + specials.
Instance methods
4Return a copy of the codepoint → id map.
Return the codepoint string for token_id or None.
Persist the vocab as vocab.txt + unified tokenizer.json.
Parameters
directorystrOutput directory (created if missing).
Build vocab by collecting unique codepoints in insertion order.
Stops when vocab_size distinct codepoints have been
observed.
Parameters
corpusiterable of strEach item is one document. Generators are consumed once.
vocab_sizeint= 1000Maximum number of distinct codepoints to retain.