Contrastive Language-Image Pre-training.
Parameters
configCLIPConfigAttributes
visual_VisionTransformertextual_TextTransformerlogit_scalenn.ParameterNotes
Reference: Radford et al., "Learning Transferable Visual Models From Natural Language Supervision", ICML, 2021 (arXiv:2103.00020).
The two towers never meet except in the dot product, so they can be
run independently — encode_image and encode_text are
the useful entry points for retrieval, where one side is indexed once
and the other is queried repeatedly.
Examples
>>> import lucid
>>> from lucid.models.multimodal.clip import CLIPModel, CLIPConfig
>>> model = CLIPModel(CLIPConfig(image_size=32, patch_size=16, vision_layers=1,
... vision_width=32, vision_heads=2,
... context_length=8, vocab_size=64, text_width=32,
... text_heads=2, text_layers=1, embed_dim=16)).eval()
>>> images = lucid.randn((2, 3, 32, 32))
>>> captions = lucid.zeros((2, 8), dtype=lucid.int64)
>>> out = model(images, captions)
>>> out.logits_per_image.shape
(2, 2)Used by 2
Constructors
1Properties
1The temperature as a multiplier — exp(logit_scale), capped.
Returns
TensorA scalar in (0, logit_scale_max].
Notes
The cap is the paper's: the scale is "clipped to prevent scaling the logits by more than 100". Applying it here rather than in a training loop means every consumer gets it, including one that writes its own loop — which is where the published implementation puts it and therefore where it is easy to omit.
Instance methods
3Embed images into the joint space, L2-normalised.
Parameters
pixel_valuesTensor(B, 3, image_size, image_size).Returns
Tensor(B, embed_dim) with unit rows.
Examples
>>> import lucid
>>> from lucid.models.multimodal.clip import CLIPConfig, CLIPModel
>>> config = CLIPConfig(image_size=32, patch_size=16, vision_layers=1,
... vision_width=32, vision_heads=2, context_length=8,
... vocab_size=64, text_width=32, text_heads=2,
... text_layers=1, embed_dim=16)
>>> model = CLIPModel(config).eval()
>>> embeds = model.encode_image(lucid.randn((3, 3, 32, 32)))
>>> embeds.shape
(3, 16)
The rows are unit length, so the Gram matrix is already cosine
similarity — each image scores exactly 1 against itself.
>>> gram = embeds @ embeds.T
>>> [round(gram[i, i].item(), 4) for i in range(3)]
[1.0, 1.0, 1.0]Embed captions into the joint space, L2-normalised.
Parameters
input_idsTensor(B, context_length).Returns
Tensor(B, embed_dim) with unit rows.
Examples
>>> import lucid
>>> from lucid.models.multimodal.clip import CLIPConfig, CLIPModel
>>> config = CLIPConfig(image_size=32, patch_size=16, vision_layers=1,
... vision_width=32, vision_heads=2, context_length=8,
... vocab_size=64, text_width=32, text_heads=2,
... text_layers=1, embed_dim=16)
>>> model = CLIPModel(config).eval()
The feature is read at [EOS] — the highest id, 63 in this
64-token vocabulary — through a causal tower, so nothing after the
sentinel can move it. These two captions differ only there.
>>> ids = lucid.tensor([[62, 5, 7, 63, 0, 0, 0, 0],
... [62, 5, 7, 63, 9, 9, 9, 9]], dtype=lucid.int64)
>>> embeds = model.encode_text(ids)
>>> embeds.shape
(2, 16)
>>> bool(lucid.allclose(embeds[0], embeds[1], atol=1e-6))
True
Scoring against images is a dot product and the temperature, which
is all forward does with the two embeddings.
>>> images = lucid.randn((2, 3, 32, 32))
>>> sims = model.encode_image(images) @ embeds.T
>>> out = model(images, ids)
>>> bool(lucid.allclose(out.logits_per_image, model.scale * sims, atol=1e-5))
Trueforward(pixel_values: Tensor, input_ids: Tensor, return_loss: bool = False)Score every image against every caption in the batch.
Parameters
Returns
CLIPOutputEmbeddings, both logit matrices, and optionally the loss.