class

SwinTransformer

extendsPretrainedModelBackboneMixin

SwinTransformer(config: SwinConfig)

source edit

Swin Transformer backbone (Liu et al., 2021).

A Swin Transformer first splits the input image into non-overlapping $4 \times 4$ patches via a strided convolution and then progresses through four hierarchical stages. Each stage stacks pairs of Swin blocks alternating between window attention (W-MSA) and shifted-window attention (SW-MSA), with patch-merging downsamplers between stages that halve spatial resolution and double channel width:

\begin{aligned} \hat{z}^{l} &= \mathrm{W\text{-}MSA}(\mathrm{LN}(z^{l-1})) + z^{l-1}, \\ z^{l} &= \mathrm{MLP}(\mathrm{LN}(\hat{z}^{l})) + \hat{z}^{l}, \\ \hat{z}^{l+1} &= \mathrm{SW\text{-}MSA}(\mathrm{LN}(z^{l})) + z^{l}, \\ z^{l+1} &= \mathrm{MLP}(\mathrm{LN}(\hat{z}^{l+1})) + \hat{z}^{l+1}. \end{aligned}

Window attention partitions the feature map into non-overlapping $M \times M$ windows and runs self-attention inside each, making the per-image cost linear in $HW$ . The shifted variant cycles each window by $\lfloor M/2 \rfloor$ pixels so that information flows between neighbouring windows across pairs of blocks. A learnable relative position bias is added inside the softmax of every window attention.

forward_features returns a global-average-pooled $(B, 8C)$ feature where $C = \texttt{embed\_dim}$ . Use this backbone when you need features rather than logits — e.g. transfer learning, dense prediction, or contrastive pretraining. For end-to-end image classification, use SwinTransformerForImageClassification instead.

Parameters

configSwinConfig

Frozen dataclass specifying image_size, patch_size, embed_dim, depths, num_heads, window_size, mlp_ratio, dropout, attention_dropout, drop_path_rate, and in_channels. See SwinConfig.

Attributes

patch_embed_PatchEmbed

Strided patch-extraction convolution + LayerNorm.

stagesnn.ModuleList

Four hierarchical _SwinStage modules, each containing a stack of Swin blocks and (for the first three) a patch-merging downsampler.

normnn.LayerNorm

Final LayerNorm applied to the channel-last feature map before global pooling.

avgpoolnn.AdaptiveAvgPool2d

1 \times 1

adaptive average pool used to produce the final

(B, 8C)

feature.

feature_infolist[FeatureInfo]

Four-stage feature description with per-stage channel counts

(C, 2C, 4C, 8C)

and spatial reductions

(p, 2p, 4p, 8p)

relative to the input.

Notes

Reference: Ze Liu et al., "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows", ICCV 2021, arXiv:2103.14030.

Examples

Build a Swin-Tiny backbone and run a forward pass:
>>> import lucid
>>> from lucid.models.vision.swin import SwinTransformer, SwinConfig
>>> model = SwinTransformer(SwinConfig())
>>> x = lucid.randn(1, 3, 224, 224)
>>> feat = model.forward_features(x)
>>> feat.shape                       # (B, 8 * embed_dim)
(1, 768)
>>> out = model(x)
>>> out.last_hidden_state.shape      # (B, 1, 8 * embed_dim)
(1, 1, 768)

Used by 2

Constructors

dunder

init

→None

__init__(config: SwinConfig)

source edit

Properties

prop

feature_info

→list[FeatureInfo]

feature_info: list[FeatureInfo]

source edit

Instance methods

forward

→BaseModelOutput

forward(x: Tensor)

source edit

forward_features

→Tensor

forward_features(x: Tensor)

source edit

Build a Swin-Tiny backbone and run a forward pass: >>> import lucid >>> from lucid.models.vision.swin import SwinTransformer, SwinConfig >>> model = SwinTransformer(SwinConfig()) >>> x = lucid.randn(1, 3, 224, 224) >>> feat = model.forward_features(x) >>> feat.shape # (B, 8 * embed_dim) (1, 768) >>> out = model(x) >>> out.last_hidden_state.shape # (B, 1, 8 * embed_dim) (1, 1, 768)