CvT backbone (Wu et al., 2021).
The Convolutional vision Transformer (CvT) reinjects the locality inductive bias of CNNs into ViT through two changes: (i) every stage starts with an overlapping convolutional token embedding (kernel > stride) that produces a new lower-resolution token grid, and (ii) the Q / K / V projections inside every self-attention are depthwise-separable convolutions rather than linear maps:
where is the spatially reshaped token map. When the K / V depthwise conv uses stride 2 the attention becomes strided, reducing by 4x inside attention without losing the full-resolution output for the next block. CvT drops positional embeddings entirely — locality is supplied implicitly by the convolutional projections.
forward_features returns the mean-pooled token feature
over the final stage's tokens.
Parameters
configCvTConfigdims, depths, num_heads,
embed_strides, mlp_ratio, dropout, and
in_channels. See CvTConfig.Attributes
stagesnn.ModuleList_CvTStage modules, each composed of a
convolutional token embedding plus a stack of CvT blocks.feature_infolist[FeatureInfo]config.embed_strides.Notes
Reference: Haiping Wu et al., "CvT: Introducing Convolutions to Vision Transformers", ICCV 2021, arXiv:2103.15808.
Examples
Build a CvT-13 backbone and run a forward pass:
>>> import lucid
>>> from lucid.models.vision.cvt import CvT, CvTConfig
>>> model = CvT(CvTConfig())
>>> x = lucid.randn(1, 3, 224, 224)
>>> feat = model.forward_features(x)
>>> feat.shape # (B, dims[-1])
(1, 384)