class
GPTTokenizer
extends
ByteLevelBPETokenizerGPTTokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)GPT-1 tokenizer — pure-Python reference.
A thin convenience subclass of
lucid.utils.tokenizer.ByteLevelBPETokenizer with GPT-1's
default add_prefix_space=False.
Parameters
vocabdict[str, int]Token-string → id map. Tokens are byte-mapped Unicode
strings (e.g.
"Ġworld" for world), NOT raw bytes.mergeslist[tuple[str, str]]Ordered BPE merges; index = rank, lower = applied earlier.
add_prefix_spacebool= FalsePrepend a space to the input so the first word also receives
the
Ġ byte-mapped marker. Off in GPT-1 (matches the
published vocab); on in RoBERTa.normalizerNormalizer= NoneDefault
lucid.utils.tokenizer.normalizers.NFC.GPT-1 ships no canonical special tokens; pass your own if
the downstream model expects bos / eos.
See Also
GPTTokenizerFast—C++-backed variant with identical output.- lucid.models.text.gpt2.GPT2Tokenizer—Successor with
<|endoftext|> as the default bos / eos / unk marker.