ByteLevelBPETokenizerFast
_ByteLevelDecodeMixinBPETokenizerFastByteLevelBPETokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)C++-backed byte-level BPE tokenizer.
Identical configuration semantics to
ByteLevelBPETokenizer; uses the engine BPE binding
for the merge hot loop and the same ByteLevel
pre-tokenizer in Python.
Encode outputs are bit-identical to the pure-Python flavour for
the same vocab + same merges.
Parameters
SameSee Also
ByteLevelBPETokenizer—Pure-Python reference; same vocab format.
Used by 3
Constructors
1__init__
→None__init__(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)Construct a C++-backed byte-level BPE tokenizer.
Parameters
vocabdict[str, int]mergeslist of (str, str)add_prefix_spacebool= FalseFalse; RoBERTa default True).normalizer(Normalizer or None, optional, keyword - only)= NoneNFC.Notes
Fixes the pre-tokenizer to ByteLevel and forwards to
BPETokenizerFast.__init__ — which constructs the C++
BPE backend and caches it on _cpp.
Properties
1Class methods
1from_file(directory: str, add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)Identical loader to ByteLevelBPETokenizer.from_file;
the only difference is the returned class (and hence the
encode backend).
Parameters
directorystradd_prefix_spacebool= Falsenormalizer(Normalizer or None, optional, keyword - only)= NoneByteLevelBPETokenizer.from_file.ByteLevelBPETokenizer.from_file.Returns
ByteLevelBPETokenizerFastA new C++-backed tokenizer populated from disk.
Raises
FileNotFoundErrortokenizer.json nor the legacy pair exists.Instance methods
1Re-train in C++ via the parent
BPETokenizerFast.train.
The ByteLevel pre-tokenizer normalizes every byte to its printable form before the corpus reaches the C++ training loop, so the resulting vocab is in byte-mapped form (Hugging Face GPT-2 compatible).
Note: the C++ BPE::train does its OWN whitespace split
on the raw input, so we pre-encode the corpus through the
ByteLevel pre-tokenizer here in Python before handing off —
otherwise the training would learn merges on raw bytes
rather than byte-mapped codepoints.
Parameters
corpusiterable of strvocab_sizeint= 50 000