class
GPT2Tokenizer
extends
ByteLevelBPETokenizerGPT2Tokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)GPT-2 tokenizer — pure-Python reference.
A thin convenience subclass of
lucid.utils.tokenizer.ByteLevelBPETokenizer with
<|endoftext|> pre-registered as the bos / eos / unk marker.
Parameters
vocabdict[str, int]Token-string → id map (byte-mapped Unicode keys).
mergeslist[tuple[str, str]]Ordered BPE merges; index = rank.
add_prefix_spacebool= FalsePrepend a space so the first word receives the
Ġ marker.
Off in GPT-2 (matches the published vocab); on in RoBERTa.normalizerNormalizer= NoneDefault
lucid.utils.tokenizer.normalizers.NFC.Override the default GPT-2 registry (single
<|endoftext|>
shared across bos / eos / unk).See Also
GPT2TokenizerFast—C++-backed variant with identical output.- lucid.models.text.gpt.GPTTokenizer—Predecessor without an
end-of-text marker.