CLIP

CLIP learns a joint embedding space for images and text by contrasting the pairings that actually occurred in a batch against the ones that did not. Given $N$ image-caption pairs, an image tower $f$ and a text tower $g$ produce $N$ embeddings each; the objective maximises the cosine similarity of the $N$ matched pairs and minimises it for the $N^2 - N$ mismatched ones. Writing $u_i = f(I_i)/\|f(I_i)\|$ and $v_j = g(T_j)/\|g(T_j)\|$, the logits are

and the loss is a cross-entropy over rows and columns alike,

The symmetry matters: a one-sided loss lets one tower collapse onto the other's geometry, because only one direction is ever scored.

The scalar $t$ is learned rather than fixed. It is stored in log space and exponentiated, so the optimiser moves it multiplicatively and it cannot become negative — a temperature is a scale, and gradient descent on the scale itself would happily cross zero. The paper initialises it at the equivalent of $\tau = 0.07$ and caps the resulting multiplier at 100, which keeps the softmax from saturating early in training when the embeddings are still arbitrary.

What makes the representation transferable is that the text tower is an open-ended classifier. A fixed-label model can only answer with the classes it was fitted to; CLIP answers with whatever captions it is given, so a new task is posed by writing its labels as sentences rather than by training a new head. This is why the prompt matters at inference time and why the paper reports prompt ensembling as an accuracy lever rather than a curiosity.