module

multimodal

0 members

lucid.models.multimodal

Models whose input spans more than one modality.

The domain exists because the alternative is a lie about where a model belongs. A CLIP filed under vision would hide a text Transformer that is half its parameters and all of its transfer story; filed under text it would hide the image tower. The zoo classifies by structure, and two encoders trained to land in one representation are a structure neither single-modality domain describes.

Concrete families live in sub-packages (clip). What is exported here is the base config — the arrangement lucid.models.generative and lucid.models.text already use.

Nothing else is here yet, and that is the rule rather than an accident: a module at this level is for what several families share. CLIP's Transformer blocks are generic enough to belong here eventually and sit in clip/_towers.py until a second family wants them, because a shared module with one consumer is an abstraction guessed rather than observed.

Model Families

family

CLIP

19 memberslucid.models.multimodal.clip

Image Classification

Radford, Alec, et al. "Learning Transferable Visual Models From Natural Language Supervision." ICML, 2021, pp. 8748–8763.

CLIP learns a joint embedding space for images and text by contrasting the pairings that actually occurred in a batch against the ones that did not. Given $N$ image-caption pairs, an image tower $f$ and a text tower $g$ produce $N$ embeddings each; the objective maximises the cosine similarity of the $N$ matched pairs and minimises it for the $N^2 - N$ mismatched ones. Writing $u_i = f(I_i)/\|f(I_i)\|$ and $v_j = g(T_j)/\|g(T_j)\|$ , the logits are