data
CLIPOutput
extends
ModelOutputCLIPOutput(image_embeds: Tensor, text_embeds: Tensor, logits_per_image: Tensor, logits_per_text: Tensor, loss: Tensor | None = None)What CLIPModel returns.
Attributes
image_embeds, text_embedsTensor(B, embed_dim), L2-normalised — so a dot product between them
is a cosine similarity and nothing further is needed to compare
them.logits_per_imageTensor(B_image, B_text), scaled by the learned temperature.logits_per_textTensorThe transpose of
logits_per_image, carried explicitly because
the symmetric loss reads both and transposing at the call site is
where the two directions get accidentally tied.lossTensor or NoneThe symmetric contrastive loss, present only when asked for.
Examples
>>> import lucid
>>> from lucid.models.multimodal.clip._model import CLIPOutput
>>> out = CLIPOutput(
... image_embeds=lucid.zeros(2, 512),
... text_embeds=lucid.zeros(3, 512),
... logits_per_image=lucid.zeros(2, 3),
... logits_per_text=lucid.zeros(3, 2),
... )
>>> out.logits_per_image.shape, out.logits_per_text.shape
((2, 3), (3, 2))
Both directions are carried because the contrastive loss reads the
matrix by rows and by columns, and one is not the other's transpose
once the temperature has been applied.