class

FasterRCNNForObjectDetection

extendsObjectDetectionModel

FasterRCNNForObjectDetection(config: FasterRCNNConfig)

source edit

Faster R-CNN with a ResNet-50-FPN backbone (Ren et al., NeurIPS 2015).

The two-stage anchor-based detector in its modern reference configuration: a ResNet-50 trunk with frozen batch-norm feeds a Feature Pyramid Network, a Region Proposal Network emits per-image proposals from five pyramid levels, and a Fast R-CNN-style RoI head classifies + refines RoI-aligned crops. The submodule layout mirrors the reference detector so the COCO box AP 37.0 checkpoint loads strict and reproduces inference.

Parameters

configFasterRCNNConfig

Frozen architecture spec. Use the faster_rcnn_resnet50_fpn factory for the COCO-pretrained configuration (num_classes = 91).

Attributes

configFasterRCNNConfig

Stored copy of the config that built this model.

backbone_BackboneWithFPN

ResNet-50 body + fpn producing five feature maps [P2, P3, P4, P5, pool] at strides 4/8/16/32/64.

rpn_RegionProposalNetwork

Proposal head shared across all pyramid levels.

roi_heads_RoIHeads

box_head (TwoMLPHead) + box_predictor (FastRCNNPredictor).

Notes

See Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NeurIPS 2015 (arXiv:1506.01497) and Lin et al., "Feature Pyramid Networks for Object Detection", CVPR 2017. The model expects an already resized + normalised image batch; final detections come from postprocess.

Examples

>>> import lucid
>>> from lucid.models.vision.faster_rcnn import faster_rcnn_resnet50_fpn
>>> model = faster_rcnn_resnet50_fpn().eval()
>>> x = lucid.randn(1, 3, 224, 224)
>>> out = model(x)
>>> out.logits.shape[-1]   # num_classes
91
>>> dets = model.postprocess(out, image_sizes=[(224, 224)])
>>> sorted(dets[0].keys())
['boxes', 'labels', 'scores']

Used by 3

Constructors

dunder

init

→None

__init__(config: FasterRCNNConfig)

source edit

Instance methods

forward

→ObjectDetectionOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, proposals: list[Tensor] | None = None)

source edit

Run Faster R-CNN on a (pre-processed) image batch.

Parameters

xTensor

(B, C, H, W) resized + normalised image batch.

targetslist[dict[str, Tensor]] | None= None

Optional per-image {"boxes": (M, 4) xyxy pixel boxes, "labels": (M,) class ids in 1..K-1}. When given, the four-term training loss is computed and the box head runs on a sampled minibatch rather than on every proposal.

proposalslist[Tensor] | None= None

Optional precomputed per-image proposals. When None the RPN generates them.

Returns

ObjectDetectionOutput

ObjectDetectionOutput with raw RoI-head outputs: logits : (Σ proposals, num_classes) class logits. pred_boxes : (Σ proposals, num_classes, 4) per-class boxes. proposals : per-image proposal tensors. loss : scalar sum of the four terms, or None.

Raises

ValueError

If targets is given but the RPN did not run, since the RPN half of the loss would then be undefined.

postprocess

→list of dict

postprocess(output: ObjectDetectionOutput, image_sizes: list[tuple[int, int]] | None = None, proposals: list[Tensor] | None = None)

source edit

Softmax → per-class score filter → per-class NMS → top-k.

Mirrors the reference postprocess_detections: drops the background class (slot 0), score-thresholds, removes empty boxes, runs per-class NMS, and keeps the top max_detections scores.

Parameters

outputObjectDetectionOutput

Raw RoI-head outputs from forward.

image_sizeslist of (H, W)= None

Per-image extent. A batch of differently-sized images is padded to one canvas, and decoding clips to that canvas — so without this a detection could legally extend into another image's padding. When given, image i's boxes are re-clipped to image_sizes[i].

proposalslist of Tensor= None

Per-image proposals; falls back to output.proposals.

Returns

list of dict

One dict per image with "boxes" (D, 4), "scores" (D,), "labels" (D,) int64.

Examples

>>> import lucid
>>> from lucid.models import create_model
>>> lucid.manual_seed(0)
>>> model = create_model(
...     "faster_rcnn_resnet50_fpn", num_classes=3,
...     backbone_layers=(1, 1, 1, 1), fpn_out_channels=32,
...     roi_representation_size=64, rpn_post_nms_top_n=10,
... ).eval()
>>> out = model(lucid.randn((1, 3, 64, 64)))
>>> det = model.postprocess(out)[0]
>>> sorted(det)
['boxes', 'labels', 'scores']
The background slot never comes back, and detections are ranked by
score.
>>> bool((det["labels"] >= 1).all())
True
>>> scores = det["scores"]
>>> bool((scores[:-1] >= scores[1:]).all())
True
Decoding clips to the 64-pixel canvas the batch ran on.  Say the
real image is 48 tall and 40 wide — as it would be inside a padded
batch — and boxes that reached into the padding are pulled back.
>>> xs = det["boxes"][:, 0::2]  # x1 and x2 of each xyxy box
>>> bool((xs <= 40).all())
False
>>> clipped = model.postprocess(out, image_sizes=[(48, 40)])[0]["boxes"]
>>> xs, ys = clipped[:, 0::2], clipped[:, 1::2]
>>> bool((xs <= 40).all()), bool((ys <= 48).all())
(True, True)

>>> import lucid >>> from lucid.models.vision.faster_rcnn import faster_rcnn_resnet50_fpn >>> model = faster_rcnn_resnet50_fpn().eval() >>> x = lucid.randn(1, 3, 224, 224) >>> out = model(x) >>> out.logits.shape[-1] # num_classes 91 >>> dets = model.postprocess(out, image_sizes=[(224, 224)]) >>> sorted(dets[0].keys()) ['boxes', 'labels', 'scores']

forward

→ObjectDetectionOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, proposals: list[Tensor] | None = None)

source edit

Run Faster R-CNN on a (pre-processed) image batch.

Parameters

xTensor

(B, C, H, W) resized + normalised image batch.

targetslist[dict[str, Tensor]] | None= None

proposalslist[Tensor] | None= None

Optional precomputed per-image proposals. When None the RPN generates them.

Returns

ObjectDetectionOutput

Raises

ValueError

If targets is given but the RPN did not run, since the RPN half of the loss would then be undefined.

>>> import lucid >>> from lucid.models import create_model >>> lucid.manual_seed(0) >>> model = create_model( ... "faster_rcnn_resnet50_fpn", num_classes=3, ... backbone_layers=(1, 1, 1, 1), fpn_out_channels=32, ... roi_representation_size=64, rpn_post_nms_top_n=10, ... ).eval() >>> out = model(lucid.randn((1, 3, 64, 64))) >>> det = model.postprocess(out)[0] >>> sorted(det) ['boxes', 'labels', 'scores'] The background slot never comes back, and detections are ranked by score. >>> bool((det["labels"] >= 1).all()) True >>> scores = det["scores"] >>> bool((scores[:-1] >= scores[1:]).all()) True Decoding clips to the 64-pixel canvas the batch ran on. Say the real image is 48 tall and 40 wide — as it would be inside a padded batch — and boxes that reached into the padding are pulled back. >>> xs = det["boxes"][:, 0::2] # x1 and x2 of each xyxy box >>> bool((xs <= 40).all()) False >>> clipped = model.postprocess(out, image_sizes=[(48, 40)])[0]["boxes"] >>> xs, ys = clipped[:, 0::2], clipped[:, 1::2] >>> bool((xs <= 40).all()), bool((ys <= 48).all()) (True, True)