class

GPT2TokenizerFast

extendsByteLevelBPETokenizerFast

GPT2TokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

GPT-2 tokenizer — C++-backed.

Bit-identical to GPT2Tokenizer. The BPE merge loop runs in C++ via the engine-side BPE binding for production throughput.

Constructor parameters mirror GPT2Tokenizer — see that class for the full reference.

Used by 1

lucid.models.text.gpt2

Constructors

dunder

init

→None

__init__(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit