class

GPTTokenizerFast

extendsByteLevelBPETokenizerFast

GPTTokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

GPT-1 tokenizer — C++-backed.

Bit-identical to GPTTokenizer. The BPE merge loop runs in C++ via the engine-side BPE binding for production throughput.

Constructor parameters mirror GPTTokenizer — see that class for the full reference.

Used by 1

lucid.models.text.gpt

Constructors

dunder

init

→None

__init__(vocab: dict[str, int], merges: list[tuple[str, str]], add_prefix_space: bool = False, normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit