class

DETRForObjectDetection

extendsPretrainedModel

DETRForObjectDetection(config: DETRConfig)

source edit

DETR end-to-end object detector (Carion et al., ECCV 2020).

The first detection model to formulate object detection as a direct set prediction problem, eliminating anchors and NMS. A ResNet backbone produces a stride-32 feature map $(B, C, H', W')$ that is projected to $d_\mathrm{model}$ channels, flattened to a sequence, and processed by a stack of transformer encoder layers. A learned set of $N$ object queries (default 100) is then decoded against this memory by the transformer decoder, with each query producing one class logit vector (K+1, including "no-object") and one box prediction $(c_x, c_y, w, h)$ normalised to $[0, 1]$ .

During training, predictions and ground-truth boxes are matched by the Hungarian algorithm under a cost combining class probability and box L1 + GIoU, and the same combination defines the per-pair loss.

Parameters

configDETRConfig

Frozen architecture spec. Use detr_resnet50 / detr_resnet101 for the paper-cited variants.

Attributes

configDETRConfig

Stored copy of the config that built this model.

backbone_ResNetC5

ResNet trunk through stage 5 (frozen BN), producing a stride-32 feature map.

input_projnn.Conv2d

1x1 conv projecting backbone output to d_model.

query_embednn.Embedding

Learned object queries of shape (num_queries, d_model).

transformer_DETRTransformer

Reference DETR transformer (encoder.layers.{N} with no final encoder norm; decoder.layers.{N} + a final decoder.norm), with per-layer positional re-injection.

class_embednn.Linear

Per-query class head with output dim num_classes + 1 (the extra "no-object" slot is essential for matching).

bbox_embed_MLP

Three-layer MLP producing per-query 4-D box predictions; the final activation is a sigmoid so boxes lie in

[0, 1]

Notes

See Carion et al., "End-to-End Object Detection with Transformers", ECCV 2020 (arXiv:2005.12872). The bipartite Hungarian assignment finds the permutation $\hat{\sigma} \in \mathfrak{S}_N$ that minimises

\hat{\sigma} = \arg\min_{\sigma \in \mathfrak{S}_N} \sum_{i=1}^{N} \mathcal{L}_\mathrm{match}\bigl(y_i, \hat{y}_{\sigma(i)}\bigr),

with the pair cost

\mathcal{L}_\mathrm{match}(y, \hat{y}) = -\mathbb{1}_{c \ne \varnothing}\,\hat{p}(c) + \mathbb{1}_{c \ne \varnothing}\, \bigl(\lambda_{\mathrm{L_1}}\|b - \hat{b}\|_1 + \lambda_\mathrm{GIoU}\, (1 - \mathrm{GIoU}(b, \hat{b}))\bigr).

The final loss reuses the same per-pair terms, with "no-object" matches penalised at reduced weight. Removing NMS / anchors makes DETR conceptually simple but training-data hungry — the paper reports needing 500 epochs to fully converge on COCO.

Examples

Inference returns class logits and normalised cxcywh boxes for every
query:
>>> import lucid
>>> from lucid.models.vision.detr import detr_resnet50
>>> model = detr_resnet50()
>>> x = lucid.randn(1, 3, 800, 800)
>>> out = model(x)
>>> out.logits.shape   # (B, num_queries, K + 1)
(1, 100, 81)
>>> out.pred_boxes.shape
(1, 100, 4)
Training pass with normalised xyxy ground-truth boxes (Hungarian
matching computes the set loss):
>>> targets = [{
...     "boxes":  lucid.tensor([[0.1, 0.1, 0.5, 0.5]]),
...     "labels": lucid.tensor([3], dtype=lucid.int64),
... }]
>>> out = model(x, targets=targets)
>>> out.loss.shape
()

Used by 2

Constructors

dunder

init

→None

__init__(config: DETRConfig)

source edit

Instance methods

forward

→ObjectDetectionOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None)

source edit

Run DETR.

Parameters

xTensor

(B, C, H, W) image batch.

targetslist[dict[str, Tensor]] | None= None

Optional training targets (list of dicts with "boxes" (xyxy normalised) and "labels").

Returns

ObjectDetectionOutput

ObjectDetectionOutput: logits : (B, N, K+1) raw class logits. pred_boxes: (B, N, 4) cxcywh normalised boxes. loss : set-prediction loss when targets provided.

postprocess

→list[dict[str, Tensor]]

postprocess(output: ObjectDetectionOutput, image_sizes: list[tuple[int, int]])

source edit

Filter queries by score threshold and convert to pixel coordinates.

Parameters

outputObjectDetectionOutput

Forward pass output.

image_sizeslist[tuple[int, int]]

List of (H, W) per image.

Returns

list[dict[str, Tensor]]

List of per-image result dicts with "boxes" (xyxy pixels),

Inference returns class logits and normalised cxcywh boxes for every query: >>> import lucid >>> from lucid.models.vision.detr import detr_resnet50 >>> model = detr_resnet50() >>> x = lucid.randn(1, 3, 800, 800) >>> out = model(x) >>> out.logits.shape # (B, num_queries, K + 1) (1, 100, 81) >>> out.pred_boxes.shape (1, 100, 4) Training pass with normalised xyxy ground-truth boxes (Hungarian matching computes the set loss): >>> targets = [{ ... "boxes": lucid.tensor([[0.1, 0.1, 0.5, 0.5]]), ... "labels": lucid.tensor([3], dtype=lucid.int64), ... }] >>> out = model(x, targets=targets) >>> out.loss.shape ()