ByteTokenizerFast
TokenizerByteTokenizerFast(special_tokens: SpecialTokens | None = None)C++-backed byte tokenizer; bit-identical to ByteTokenizer.
Same semantics as ByteTokenizer, hot loop in C++.
Speedup vs the Python flavour is modest (the algorithm is already
a trivial UTF-8 encode loop) but the API uniformity matters: any
code that swaps XxxTokenizer ↔ XxxTokenizerFast works
consistently across the family.
Parameters
ByteTokenizer for the
constraints on default-vocab compatibility.Used by 1
Constructors
1Construct a C++-backed byte tokenizer.
Parameters
Notes
Instantiates the C++ ByteTokenizer backend; the canonical
256-entry Python vocab is also cached for get_vocab /
id_to_token parity.
Properties
2Class methods
1from_file(directory: str, special_tokens: SpecialTokens | None = None)Load from directory (only the special-token map is read).
Parameters
directorystrsave.Returns
ByteTokenizerFastNew instance with the loaded special-token registry.
Instance methods
4Return the canonical 256-entry byte → id map.
Return the Latin-1 char for token_id or None.
Persist as tokenizer.json + special_tokens_map.json.
Parameters
directorystrNo-op — the byte vocab is fixed at 256. See
ByteTokenizer.train.