class

EfficientFormer

extendsPretrainedModelBackboneMixin

EfficientFormer(config: EfficientFormerConfig)

source edit

EfficientFormer backbone (Li et al., 2022).

EfficientFormer is a mobile-grade vision backbone designed so that its on-device latency, rather than its FLOP count, matches MobileNet on the same hardware while retaining transformer-level accuracy. The trunk has four stages preceded by a two stride-2 convolutional stem. Stages 1-3 are MetaFormer-style pooling blocks operating in $(B, C, H, W)$ layout — no reshape, no self-attention:

\begin{aligned} x &\leftarrow x + \gamma_1 \odot \bigl( \mathrm{AvgPool}_{3 \times 3}(x) - x\bigr), \\ x &\leftarrow x + \gamma_2 \odot \mathrm{ConvMLP}(x), \end{aligned}

with $\gamma_1, \gamma_2$ initialised to $10^{-5}$ (CaiT-style layer scale). The last stage switches its trailing $\texttt{num\_vit}$ blocks to standard multi-head self-attention with a learned relative-position bias on the now-tiny token grid.

forward_features returns the mean-pooled $(B, \texttt{embed\_dims[-1]})$ feature.

Parameters

configEfficientFormerConfig

Frozen dataclass specifying depths, embed_dims, mlp_ratios, num_vit, drop_path_rate, layer_scale_init, in_channels, and num_classes. See EfficientFormerConfig.

Attributes

stem_Stem

Two stride-2

3 \times 3

convolutions reducing the input by 4x.

stagesnn.ModuleList

Four stages — three pooling stages and one mixed pooling/attention stage; stages 1-3 carry a downsample.

normnn.LayerNorm

Final LayerNorm applied to the channel-last token sequence.

feature_infolist[FeatureInfo]

Four-stage feature description with reductions

(4, 8, 16, 32)

Notes

Reference: Yanyu Li et al., "EfficientFormer: Vision Transformers at MobileNet Speed", NeurIPS 2022, arXiv:2206.01191.

Examples

Build an EfficientFormer-L1 backbone and run a forward pass:
>>> import lucid
>>> from lucid.models.vision.efficientformer import (
...     EfficientFormer, EfficientFormerConfig,
... )
>>> model = EfficientFormer(EfficientFormerConfig())
>>> x = lucid.randn(1, 3, 224, 224)
>>> feat = model.forward_features(x)
>>> feat.shape                       # (B, embed_dims[-1])
(1, 448)

Used by 2

Constructors

dunder

init

→None

__init__(config: EfficientFormerConfig)

source edit

Properties

prop

feature_info

→list[FeatureInfo]

feature_info: list[FeatureInfo]

source edit

Instance methods

forward

→BaseModelOutput

forward(x: Tensor)

source edit

forward_features

→Tensor

forward_features(x: Tensor)

source edit

Build an EfficientFormer-L1 backbone and run a forward pass: >>> import lucid >>> from lucid.models.vision.efficientformer import ( ... EfficientFormer, EfficientFormerConfig, ... ) >>> model = EfficientFormer(EfficientFormerConfig()) >>> x = lucid.randn(1, 3, 224, 224) >>> feat = model.forward_features(x) >>> feat.shape # (B, embed_dims[-1]) (1, 448)