CvT

The Convolutional vision Transformer (CvT) reinjects the locality inductive bias of CNNs into ViT through two changes: (i) replacing the non-overlapping patch embedding by overlapping convolutional token embedding, and (ii) replacing the linear projections inside self-attention with depthwise-separable convolutional projections. CvT also drops positional embeddings entirely — locality is supplied implicitly by the convolutional projections.

Convolutional token embedding is performed at the start of each of the three stages: a strided convolution $\mathrm{Conv}_{k \times k, \mathrm{stride}=s}$ produces a new token grid of reduced spatial resolution and increased channel width, yielding a Swin-like hierarchical pyramid. Inside every block the queries / keys / values are computed with depthwise convolutions rather than linear maps:

where $x_{2D}$ is the spatially reshaped token map. When the key/value DWConv uses stride 2 the attention becomes strided, reducing $N$ by 4 inside attention without losing the full-resolution output for the next block. Variants CvT-13 / 21 / 24 scale depth and width along this three-stage convolutional transformer skeleton.