ByteLevelBPETokenizer
_ByteLevelDecodeMixinBPETokenizerByteLevelBPETokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)Pure-Python byte-level BPE (GPT-2 / RoBERTa convention).
Subclasses BPETokenizer and swaps in a
lucid.utils.tokenizer._pre_tokenizers.ByteLevel
pre-tokenizer so the underlying BPE algorithm sees a printable
Unicode alphabet of size 256 instead of raw bytes. The GPT-2
byte-to-unicode mapping shifts the unprintable bytes
(0x00–0x20, 0x7F–0xA0) into a safe Unicode block
so every byte sequence is representable and no UNK token is
needed.
The merge table + training loop are inherited unchanged; the
only differences are the pre-tokenizer (encode side) and the
byte-mapping reversal in ByteLevelBPETokenizer.decode
(decode side).
For production / latency-sensitive use, prefer
ByteLevelBPETokenizerFast.
Parameters
vocabdict[str, int]"Ġworld" for world), not
raw bytes.mergeslist[tuple[str, str]]add_prefix_spacebool= FalseĠ prefix).
GPT-2 default is False; RoBERTa default is True.normalizerNormalizer= Nonelucid.utils.tokenizer._normalizers.NFC (matches
GPT-2).lucid.utils.tokenizer.SpecialTokens.Notes
Any published Hugging Face GPT-2 / RoBERTa / BART / distilGPT
checkpoint loads without modification via
ByteLevelBPETokenizer.from_pretrained.
See Also
ByteLevelBPETokenizerFast—C++-backed equivalent with the same API.BPETokenizer—Classical (non byte-level) BPE base.
Used by 3
Constructors
1__init__
→None__init__(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)Construct a pure-Python byte-level BPE tokenizer.
Parameters
vocabdict[str, int]mergeslist of (str, str)add_prefix_spacebool= FalseFalse,
RoBERTa: True).normalizer(Normalizer or None, optional, keyword - only)= NoneNFC.Notes
The pre-tokenizer is fixed to ByteLevel with the
chosen add_prefix_space — that's the whole point of
byte-level BPE. Forwards everything else to
BPETokenizer.__init__.
Properties
1Class methods
1from_file(directory: str, add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)Load from a Hugging Face-compatible directory.
Same on-disk format as BPETokenizer: either the
legacy vocab.json + merges.txt pair OR the unified
tokenizer.json. When the unified file is present, it
wins and its model.add_prefix_space overrides the
add_prefix_space argument.
Parameters
directorystradd_prefix_spacebool= Falsenormalizer(Normalizer or None, optional, keyword - only)= Nonespecial_tokens_map.json when omitted.Returns
ByteLevelBPETokenizerA new tokenizer populated from disk.
Raises
FileNotFoundErrortokenizer.json nor the legacy pair exists
in directory.Instance methods
1Re-train this tokenizer from scratch on corpus.
Delegates to the parent BPETokenizer.train. The
configured ByteLevel pre-tokenizer means the
algorithm sees byte-mapped chunks throughout training, so
the resulting vocab + merges are in GPT-2 byte-mapped form
(compatible with Hugging Face GPT-2 / RoBERTa export).
Parameters
corpusiterable of strvocab_sizeint= 50 000