CLIPConfig
MultimodalModelConfigCLIPConfig(embed_dim: int = 512, act_fn: MultimodalActivation = 'quick_gelu', image_size: int = 224, patch_size: int = 32, vision_layers: int = 12, vision_width: int = 768, vision_heads: int = 12, context_length: int = 77, vocab_size: int = 49408, text_width: int = 512, text_heads: int = 8, text_layers: int = 12, logit_scale_init: float = 0.07, logit_scale_max: float = 100.0)Frozen configuration dataclass for every CLIP variant.
Parameters
embed_dimint= 512lucid.models.multimodal.MultimodalModelConfig
— the width of the joint image-text space both towers project
into, and the dimension a downstream consumer sees.act_fnMultimodalActivation= "quick_gelu"image_sizeint= 224336 for the
high-resolution ViT-L/14 variant.patch_sizeint= 32(image_size / patch_size) ** 2 + 1, the + 1
being the class token, so halving this quadruples the sequence.vision_layersint= 12vision_widthint= 768vision_headsint= 12vision_width / 64.context_lengthint= 77[SOS] and
[EOS] sentinels.vocab_sizeint= 49408text_widthint= 512text_headsint= 8text_layersint= 12logit_scale_initfloat= 0.07logit_scale_maxfloat= 100.0Notes
Two numbers here disagree with the paper's prose, and the implementation's are used because the released weights have those shapes:
================= ================== ==================
field paper §A official release
================= ================== ==================
context_length "capped at 76" 77
vocab_size "49,152 vocab size" 49408
================= ================== ==================
The gap in each case is the sentinels: 76 caption tokens plus
[SOS] and [EOS] does not fit in 77, so the released models
treat 77 as the whole budget; and 49408 − 49152 = 256, one slot per
byte, which is what a byte-level BPE needs to stay total.
Depth scales only in the image tower. Every variant the paper
trains keeps text_layers = 12, and for the ResNet towers it says
so outright — "we only scale the width of the model to be
proportional to the calculated increase in width of the ResNet and do
not scale the depth at all". The ViT variants follow the same rule:
ViT-L/14 widens the text tower to 768 and leaves it 12 deep.
Examples
>>> from lucid.models.multimodal.clip import CLIPConfig
>>> config = CLIPConfig()
>>> config.embed_dim, config.patch_size
(512, 32)
>>> large = CLIPConfig(embed_dim=768, patch_size=14, vision_layers=24,
... vision_width=1024, vision_heads=16,
... text_width=768, text_heads=12)
>>> large.vision_width // large.vision_heads
64Used by 3
Constructors
1__init__
→None__init__(embed_dim: int = 512, act_fn: MultimodalActivation = 'quick_gelu', image_size: int = 224, patch_size: int = 32, vision_layers: int = 12, vision_width: int = 768, vision_heads: int = 12, context_length: int = 77, vocab_size: int = 49408, text_width: int = 512, text_heads: int = 8, text_layers: int = 12, logit_scale_init: float = 0.07, logit_scale_max: float = 100.0)Initialise the tower. See the class docstring for parameters.