class

MaskFormerForSemanticSegmentation

extendsPretrainedModel

MaskFormerForSemanticSegmentation(config: MaskFormerConfig)

source edit

MaskFormer semantic-segmentation model (Cheng et al., NeurIPS 2021).

Recasts semantic segmentation as mask classification: instead of per-pixel cross-entropy, the model predicts a fixed set of $N$ (mask, class-probability) pairs, computes the per-pixel class probability as

p_\mathrm{px}(c) = \sum_{i=1}^{N} p_i(c)\, m_i(\mathrm{px}),

and supervises the set with bipartite (Hungarian) matching to ground- truth masks. This unification lets a single architecture handle semantic, instance, and panoptic segmentation with the same training objective — the breakthrough that the follow-up Mask2Former extends with masked attention.

Architecturally: a ResNet backbone -> FPN-style pixel decoder produces a per-pixel embedding of dimension mask_feature_size, and a DETR-style transformer decoder operates on $N$ query embeddings, cross-attending to the projected backbone C5 feature map; sibling heads then produce $(N, K+1)$ class logits and a 3-layer-MLP mask embedding. Each query's binary mask is recovered by a dot product with the pixel embedding.

Parameters

configMaskFormerConfig

Frozen architecture spec. Use maskformer_resnet50 / maskformer_resnet101 for the paper-cited variants (ADE20K, 150 classes).

Attributes

configMaskFormerConfig

Stored copy of the config that built this model.

pixel_level_module_PixelLevelModule

ResNet encoder + FPN pixel decoder yielding the raw C5 feature and a per-pixel embedding

(B, d, H/4, W/4)

transformer_module_TransformerModule

Learned queries + input_projection + a 6-layer DETR-style decoder.

class_predictornn.Linear

Per-query

(K + 1)

classification head (+1 for the "no object" class).

mask_embedder_MaskEmbedder

Per-query 3-layer MLP producing the dot-product mask weights.

Notes

See Cheng et al., "Per-Pixel Classification is Not All You Need for Semantic Segmentation", NeurIPS 2021 (arXiv:2107.06278). The total loss combines a CE term over query class predictions and a per-mask binary CE + dice term, summed only over the Hungarian-matched query permutation $\hat{\sigma}$ .

For semantic-segmentation inference the model drops the "no-object" slot and collapses queries by softmax-weighted summation into a standard $(B, K, H, W)$ logit map for direct argmax evaluation.

Examples

>>> import lucid
>>> from lucid.models.vision.maskformer import maskformer_resnet50
>>> model = maskformer_resnet50()
>>> x = lucid.randn(1, 3, 512, 512)
>>> out = model(x)
>>> out.logits.shape   # (B, K, H, W)
(1, 150, 512, 512)

Used by 2

Constructors

dunder

init

→None

__init__(config: MaskFormerConfig)

source edit

Instance methods

forward

→SemanticSegmentationOutput

forward(x: Tensor, targets: dict[str, Tensor] | None = None)

source edit

Run MaskFormer.

Parameters

xTensor

(B, C, H, W) image batch.

targetsdict[str, Tensor] | None= None

Optional dict with "masks" (B, H, W) integer masks.

Returns

SemanticSegmentationOutput

SemanticSegmentationOutput: logits: (B, K, H, W) per-pixel class logits (no-object slot dropped — matches the reference semantic post-processing). loss: training loss when targets provided.

>>> import lucid >>> from lucid.models.vision.maskformer import maskformer_resnet50 >>> model = maskformer_resnet50() >>> x = lucid.randn(1, 3, 512, 512) >>> out = model(x) >>> out.logits.shape # (B, K, H, W) (1, 150, 512, 512)