class

CLIPTokenizerFast

extendsCLIPTokenizer

CLIPTokenizerFast(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

CLIP's tokenizer with the merge loop running in C++.

Same vocabulary format, same preprocessing, same output as CLIPTokenizer — only faster.

Parameters

vocabdict[str, int]

Token-string → id map, over byte-mapped symbols.

mergeslist of tuple of str

Ordered BPE merges; index is the rank.

normalizerNormalizer= None

Applied before the pattern.

special_tokensSpecialTokens= None

Override the default sentinel registry.

Notes

How this is possible at all. The engine's BPE seeds one symbol per codepoint, so handing it "hello</w>" tears the marker into <, /, w, > — four symbols where the scheme needs the marker fused to the final character. That was measured, and it is why lucid.models.text.gpt.GPTTokenizerFast carries no acceleration at all.

The way through is a vocabulary rewrite rather than an engine one. Every entry ending in </w> is folded so the marker and the character before it become a single private-use codepoint, and the same fold is applied to the merge table and to each chunk before it is handed over. The engine then seeds exactly the symbols the scheme intends, applies the merges it was given, and returns ids from the unmodified id space — the fold changes only the keys, never the values, so nothing downstream sees it.

The same trick would give GPT-1 a genuine fast path; that class's note that the scheme "needs an engine change" is one option, not the only one.

Examples

A handful of tokens will not do: the constructor requires a *total*
byte-level vocabulary, for the reason given above, so the example has
to build one.  ByteLevel is where the alphabet comes from — the
same 256 symbols the pre-tokenizer maps bytes onto.
>>> from lucid.models.multimodal.clip import CLIPTokenizerFast
>>> from lucid.utils.tokenizer._pre_tokenizers import ByteLevel
>>> alphabet = [ByteLevel.encode_bytes(bytes([b])) for b in range(256)]
>>> vocab = {symbol: i for i, symbol in enumerate(alphabet)}
>>> vocab.update({s + "</w>": 256 + i for i, s in enumerate(alphabet)})
>>> vocab["ab</w>"] = 512  # what the one merge below produces
>>> vocab["<|startoftext|>"], vocab["<|endoftext|>"] = 513, 514
>>> tok = CLIPTokenizerFast(vocab=vocab, merges=[("a", "b</w>")])
>>> tok.encode("ab")
[512]
>>> tok.tokenize("ab", context_length=5)
[513, 512, 514, 0, 0]
Pass a partial vocabulary instead and the constructor raises rather
than quietly disagreeing with CLIPTokenizer, which accepts
one and substitutes UNK.

Used by 1

lucid.models.multimodal.clip

Constructors

dunder

init

→None

__init__(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)

source edit

Construct the tokenizer. See the class docstring for parameters.

A handful of tokens will not do: the constructor requires a *total* byte-level vocabulary, for the reason given above, so the example has to build one. ByteLevel is where the alphabet comes from — the same 256 symbols the pre-tokenizer maps bytes onto. >>> from lucid.models.multimodal.clip import CLIPTokenizerFast >>> from lucid.utils.tokenizer._pre_tokenizers import ByteLevel >>> alphabet = [ByteLevel.encode_bytes(bytes([b])) for b in range(256)] >>> vocab = {symbol: i for i, symbol in enumerate(alphabet)} >>> vocab.update({s + "</w>": 256 + i for i, s in enumerate(alphabet)}) >>> vocab["ab</w>"] = 512 # what the one merge below produces >>> vocab["<|startoftext|>"], vocab["<|endoftext|>"] = 513, 514 >>> tok = CLIPTokenizerFast(vocab=vocab, merges=[("a", "b</w>")]) >>> tok.encode("ab") [512] >>> tok.tokenize("ab", context_length=5) [513, 512, 514, 0, 0] Pass a partial vocabulary instead and the constructor raises rather than quietly disagreeing with CLIPTokenizer, which accepts one and substitutes UNK.