ByteTokenizer
TokenizerByteTokenizer(special_tokens: SpecialTokens | None = None)Reference (pure-Python) byte-level tokenizer.
Vocab is fixed at 256 (byte value → id == byte value). Equivalent
to ByteTokenizerFast but loops in Python — useful when
debugging downstream consumers without a C++ dependency.
Parameters
chr(b) for some b). IDs
beyond 255 are NOT supported on the default vocab — extend
the vocab manually before adding such specials.Used by 1
Constructors
1Construct a pure-Python byte tokenizer.
Parameters
Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from a directory containing special_tokens_map.json.
The vocab itself is not read from disk (always the canonical
256-entry table); we only consult special_tokens_map.json
so user-defined specials survive a round-trip.
Parameters
directorystrsave.Returns
ByteTokenizerNew instance with the loaded special-token registry.
Instance methods
4Return the canonical 256-entry byte → id map.
Inverse of the byte vocab; returns the Latin-1 char for
valid byte ids, None otherwise.
Persist as tokenizer.json only (no per-algorithm
legacy file — there's no vocab.txt convention for byte
tokenizers).
Parameters
directorystrtokenizer.json + special_tokens_map.json.No-op — the byte vocab is fixed at 256.
Parameters
corpusiterable of strvocab_sizeint= 256