module

Swin Transformer

15 members

lucid.models.vision.swin

Swin Transformer family — Liu et al., 2021.

Image Classification

Paper

Liu, Ze, et al. "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012-10022.

Overview

The Swin Transformer reintroduces the hierarchical feature pyramid of classical CNNs into a pure-transformer backbone, making it a general-purpose vision model that scales linearly with image resolution rather than quadratically (as plain ViT does). The image is first split into non-overlapping $4 \times 4$ patches and projected to $C$ channels. Four stages then follow, each halving the spatial resolution via a patch merging layer that concatenates $2 \times 2$ neighbours and projects $4C \to 2C$ , doubling the channel width — giving the canonical $C, 2C, 4C, 8C$ pyramid.

Within each stage the tokens flow through pairs of Swin blocks. The first block computes self-attention inside non-overlapping windows of size $M \times M$ (default $M = 7$ ), and the second performs shifted-window (SW-MSA) attention where the grid is cyclically shifted by $\lfloor M/2 \rfloor$ . Because each window is processed independently, the attention cost per image is $\mathcal{O}(M^2 H W)$ instead of $\mathcal{O}((HW)^2)$ , but the shift introduces cross-window connections so the receptive field still grows with depth. A learnable relative position bias $B \in \mathbb{R}^{M^2 \times M^2}$ is added inside softmax:

\mathrm{Attn}(Q, K, V) = \mathrm{softmax}\!\left( \frac{Q K^\top}{\sqrt{d}} + B \right) V.

The four canonical variants (Tiny / Small / Base / Large) scale $C$ and per-stage block counts from Table 7 of the paper, trading parameters for accuracy while reusing the same shifted-window mechanism end to end.

Classes

dataSwinConfig1 methods

Configuration dataclass for every Swin Transformer variant.

classSwinTransformer4 methods

Swin Transformer backbone (Liu et al., 2021).

classSwinTransformerForImageClassification2 methods

Swin Transformer with a linear classification head (Liu et al., 2021).

Functions

fnswin_tiny→ SwinTransformer

Swin-Tiny backbone (Liu et al., 2021).

fnswin_tiny_cls→ SwinTransformerForImageClassification

Swin-Tiny image classifier (Liu et al., 2021).

fnswin_small→ SwinTransformer

Swin-Small backbone (Liu et al., 2021).

fnswin_small_cls→ SwinTransformerForImageClassification

Swin-Small image classifier (Liu et al., 2021).

fnswin_base→ SwinTransformer

Swin-Base backbone (Liu et al., 2021).

fnswin_base_cls→ SwinTransformerForImageClassification

Swin-Base image classifier (Liu et al., 2021).

fnswin_large→ SwinTransformer

Swin-Large backbone (Liu et al., 2021).

fnswin_large_cls→ SwinTransformerForImageClassification

Swin-Large image classifier (Liu et al., 2021).

Weights

classSwinBaseWeights

Pretrained weight tags for lucid.models.swin_base_cls.

classSwinLargeWeights

Pretrained weight tags for lucid.models.swin_large_cls.

classSwinSmallWeights

Pretrained weight tags for lucid.models.swin_small_cls.

classSwinTinyWeights

Pretrained weight tags for lucid.models.swin_tiny_cls.