CLIPTokenizer
BPETokenizerCLIPTokenizer(vocab: dict[str, int], merges: list[tuple[str, str]], normalizer: Normalizer | None = None, special_tokens: SpecialTokens | None = None)CLIP's byte-level BPE with </w> word boundaries.
Parameters
vocabdict[str, int]CLIP_SOS and CLIP_EOS.mergeslist of tuple of strnormalizerNormalizer= Nonebos to
<|startoftext|> and eos / unk to <|endoftext|>.Notes
Reference: Radford et al., ICML 2021 (arXiv:2103.00020), §2.3.
tokenize is the method the model wants: it frames each
caption with the two sentinels and pads to a fixed width, which is
what makes argmax over the ids find [EOS]. encode
is inherited and does neither — it returns the bare BPE ids, which
is what a caller building their own framing wants.
Examples
>>> from lucid.models.multimodal.clip import CLIPTokenizer
>>> vocab = {"a": 0, "b</w>": 1, "ab</w>": 2,
... "<|startoftext|>": 3, "<|endoftext|>": 4}
>>> tok = CLIPTokenizer(vocab=vocab, merges=[("a", "b</w>")])
>>> tok.encode("ab")
[2]
>>> tok.tokenize("ab", context_length=5)
[3, 2, 4, 0, 0]Used by 1
Constructors
1Instance methods
2Return the BPE ids of text, unframed and unpadded.
Parameters
textstradd_special_tokensbool= False, keyword-onlyTrue. It is flipped here
because tokenize is the framing path and adding the
sentinels in both places gives a caption two [SOS],
shifting every position the model reads by one.Returns
list of intOne id per merged symbol.
Notes
The sentinels are not added here, unlike the base class's
default. Framing belongs to tokenize, which also pads —
adding them in both places is how a caption ends up with two
[SOS] and the model reads its feature one position early.
Examples
>>> from lucid.models.multimodal.clip import CLIPTokenizer
>>> vocab = {"a": 0, "b</w>": 1, "ab</w>": 2,
... "<|startoftext|>": 3, "<|endoftext|>": 4}
>>> tok = CLIPTokenizer(vocab=vocab, merges=[("a", "b</w>")])
The caption is cleaned before it is split — lowercased, whitespace
runs collapsed — and comes back bare unless the sentinels are
asked for.
>>> tok.encode("AB ab")
[2, 2]
>>> tok.encode("ab", add_special_tokens=True)
[3, 2, 4]
So decoding returns the cleaned text, not the original.
>>> tok.decode(tok.encode("AB ab"))
'ab ab'Frame a caption with the sentinels and pad it to a fixed width.
Parameters
textstrcontext_lengthint= 77Returns
list of int[SOS] … [EOS] followed by zeros.
Raises
ValueError[EOS] the model locates its feature by, so a caption
that is too long is an error rather than a shortened result.Notes
Padding is 0, which is a real token id in a CLIP vocabulary.
That is harmless because the feature is read at [EOS] and
the text tower is causal, so nothing after the sentinel can reach
the position the feature is taken from.
Examples
>>> from lucid.models.multimodal.clip import CLIPTokenizer
>>> vocab = {"a": 0, "b</w>": 1, "ab</w>": 2,
... "<|startoftext|>": 3, "<|endoftext|>": 4}
>>> tok = CLIPTokenizer(vocab=vocab, merges=[("a", "b</w>")])
>>> tok.tokenize("ab", context_length=4)
[3, 2, 4, 0]