CrossViT
21 memberslucid.models.vision.crossvitCrossViT family — backbone + image classification (Chen et al., 2021).
Chen, Chun-Fu (Richard), et al. "CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 357-366.
CrossViT explicitly models multi-scale visual information by running two parallel ViT branches with different patch sizes and fusing them through a lightweight cross-attention mechanism. A small-patch branch (patch size 12) processes a long sequence of fine-grained tokens, while a large-patch branch (patch size 16) processes a short sequence of coarse tokens. The two branches operate on different input resolutions — the small-patch branch on the full image (e.g. ), the large-patch branch on a slightly down-sampled version (e.g. ) — and each has its own class token plus learnable positional embedding.
Each branch is independently transformer-encoded for / standard self-attention blocks; the two branches then exchange information at the end of every stage via a MultiScaleBlock: the CLS token from one branch is projected into the other branch's embedding space, cross-attends to that branch's patch tokens, and is projected back. Concretely, for the small-to-large direction,
and symmetrically for large-to-small. Because cross-attention is only per direction, multi-scale fusion is essentially free relative to the per-branch self-attention cost. The complete network is such MultiScaleBlocks stacked back-to-back; the final classifier head averages the logits of the two per-branch linear layers.
The variants CrossViT-Ti / -S / -B / -9 / -15 / -18 trade depth and width along this two-branch skeleton (paper Table 2).
Classes
Functions
crossvit_9→ CrossViT8.6MCrossViT-9 backbone — embed_dims=(128, 256), depths
((1, 3, 0))×3, 4 heads. ~8.6M params (paper Table 2).
crossvit_9_cls→ CrossViTForImageClassification8.6MCrossViT-9 image classifier — embed_dims=(128, 256), depths
((1, 3, 0)) x 3, 4 heads per branch. ~8.6M params; paper
Table 2 reports 73.9% ImageNet-1k top-1 at 240x240.
crossvit_tiny→ CrossViT7MCrossViT-Ti backbone — embed_dims=(96, 192), depths
((1, 4, 0))×3, 3 heads. ~7M params (paper Table 2).
crossvit_tiny_cls→ CrossViTForImageClassification7MCrossViT-Ti image classifier (Chen et al., ICCV 2021).
crossvit_small→ CrossViT26.7MCrossViT-S backbone — embed_dims=(192, 384), depths
((1, 4, 0))×3, 6 heads. ~26.7M params (paper Table 2).
crossvit_small_cls→ CrossViTForImageClassification26.7MCrossViT-S image classifier — embed_dims=(192, 384), depths
((1, 4, 0)) x 3, 6 heads per branch. ~26.7M params; paper
Table 2 reports 81.0% ImageNet-1k top-1 at 240x240.
crossvit_base→ CrossViT105MCrossViT-B backbone — embed_dims=(384, 768), depths
((1, 4, 0))×3, 12 heads. ~105M params (paper Table 2).
crossvit_base_cls→ CrossViTForImageClassification105MCrossViT-B image classifier — embed_dims=(384, 768), depths
((1, 4, 0)) x 3, 12 heads per branch. ~105M params; paper
Table 2 reports 82.2% ImageNet-1k top-1 at 240x240.
crossvit_15→ CrossViT27.4MCrossViT-15 backbone — embed_dims=(192, 384), depths
((1, 5, 0))×3, 6 heads. ~27.4M params (paper Table 2).
crossvit_15_cls→ CrossViTForImageClassification27.4MCrossViT-15 image classifier — embed_dims=(192, 384), depths
((1, 5, 0)) x 3, 6 heads per branch. ~27.4M params; paper
Table 2 reports 81.5% ImageNet-1k top-1 at 240x240.
crossvit_18→ CrossViT43.3MCrossViT-18 backbone — embed_dims=(224, 448), depths
((1, 6, 0))×3, 7 heads. ~43.3M params (paper Table 2).
crossvit_18_cls→ CrossViTForImageClassification43.3MCrossViT-18 image classifier — embed_dims=(224, 448), depths
((1, 6, 0)) x 3, 7 heads per branch. ~43.3M params; paper
Table 2 reports 82.5% ImageNet-1k top-1 at 240x240.
Weights
CrossViT9WeightsPretrained weights for lucid.models.crossvit_9_cls.
CrossViT15WeightsPretrained weights for lucid.models.crossvit_15_cls.
CrossViT18WeightsPretrained weights for lucid.models.crossvit_18_cls.
CrossViTBaseWeightsPretrained weights for lucid.models.crossvit_base_cls.
CrossViTSmallWeightsPretrained weights for lucid.models.crossvit_small_cls.
CrossViTTinyWeightsPretrained weights for lucid.models.crossvit_tiny_cls.