class
CLIPForZeroShotImageClassification
extends
ImageClassificationModelCLIPForZeroShotImageClassification(config: CLIPConfig)CLIP posed as a classifier, by writing the labels as sentences.
Parameters
configCLIPConfigThe variant to build.
Notes
There is no head and nothing new is trained. The classes arrive as tokenised prompts and the scores are the same cosine similarities the contrastive objective was fitted on, which is the whole claim of the paper — a fixed-label model cannot be asked a question it was not trained for, and this can.
A caller that classifies many images against one label set should
embed the prompts once with CLIPModel.encode_text and reuse them;
forward re-embeds them on every call because it does not know
that the set is fixed.
Examples
>>> import lucid
>>> from lucid.models.multimodal.clip import (
... CLIPConfig, CLIPForZeroShotImageClassification)
>>> config = CLIPConfig(image_size=32, patch_size=16, vision_layers=1,
... vision_width=32, vision_heads=2, context_length=8,
... vocab_size=64, text_width=32, text_heads=2,
... text_layers=1, embed_dim=16)
>>> model = CLIPForZeroShotImageClassification(config).eval()
>>> out = model(lucid.randn((2, 3, 32, 32)), lucid.zeros((5, 8), dtype=lucid.int64))
>>> out.logits.shape
(2, 5)Used by 2
Constructors
1Instance methods
1forward(pixel_values: Tensor, prompt_ids: Tensor)Score each image against each candidate prompt.
Parameters
Returns
CLIPZeroShotOutputlogits is (B, num_prompts).