class

EfficientFormerForImageClassification

extendsPretrainedModelClassificationHeadMixin

EfficientFormerForImageClassification(config: EfficientFormerConfig)

source edit

EfficientFormer with a distilled dual classification head (Li et al., 2022).

Wraps the same trunk as EfficientFormer and adds a final LayerNorm, a mean pool over tokens, and two linear heads — the classification head and the distillation head — whose logits are averaged at inference:

\text{logits} = \tfrac{1}{2}\bigl( W_{\text{head}}\,z + b_{\text{head}} + W_{\text{dist}}\,z + b_{\text{dist}}\bigr), \qquad z = \mathrm{Mean}(\mathrm{LN}(z^{L})).

The published EfficientFormer checkpoints were trained with hard knowledge distillation (DeiT-style), so reproducing the reported top-1 accuracy requires averaging both heads.

Pass labels to forward to compute the cross-entropy loss in the same pass.

Parameters

configEfficientFormerConfig

Architecture specification. Must set num_classes to the desired number of output categories. See EfficientFormerConfig.

Attributes

stem_Stem

Two stride-2 convolutions.

stagesnn.ModuleList

Three pooling stages + one mixed pooling/attention stage.

normnn.LayerNorm

Final LayerNorm.

headnn.Linear

Classification head (num_classes, embed_dims[-1]).

head_distnn.Linear

Distillation head (num_classes, embed_dims[-1]); averaged with head at inference.

Notes

Reference: Li et al., "EfficientFormer: Vision Transformers at MobileNet Speed", NeurIPS 2022. EfficientFormer-L1 / L3 / L7 reach 79.2 / 82.4 / 83.3 % top-1 on ImageNet-1k at MobileNetV2- class on-device latency (Li et al., 2022, Table 4).

Examples

End-to-end inference with the default EfficientFormer-L1
classifier:
>>> import lucid
>>> from lucid.models.vision.efficientformer import (
...     EfficientFormerConfig, EfficientFormerForImageClassification,
... )
>>> model = EfficientFormerForImageClassification(EfficientFormerConfig())
>>> x = lucid.randn(1, 3, 224, 224)
>>> out = model(x)
>>> out.logits.shape
(1, 1000)

Used by 2

Constructors

dunder

init

→None

__init__(config: EfficientFormerConfig)

source edit

Instance methods

forward

→ImageClassificationOutput

forward(x: Tensor, labels: Tensor | None = None)

source edit

reset_classifier

→None

reset_classifier(num_classes: int)

source edit

End-to-end inference with the default EfficientFormer-L1 classifier: >>> import lucid >>> from lucid.models.vision.efficientformer import ( ... EfficientFormerConfig, EfficientFormerForImageClassification, ... ) >>> model = EfficientFormerForImageClassification(EfficientFormerConfig()) >>> x = lucid.randn(1, 3, 224, 224) >>> out = model(x) >>> out.logits.shape (1, 1000)