class

MaskRCNNForObjectDetection

extendsObjectDetectionModel

MaskRCNNForObjectDetection(config: MaskRCNNConfig)

source edit

Mask R-CNN with a ResNet-50-FPN backbone (He et al., ICCV 2017).

The two-stage instance-segmentation detector in its modern reference configuration: Faster R-CNN's ResNet-50-FPN backbone, RPN, and Fast R-CNN box head, plus a parallel FCN mask branch on the RoI heads. The submodule layout mirrors the reference detector so the COCO box AP 37.9 / mask AP 34.6 checkpoint loads strict (307 keys) and reproduces inference.

Parameters

configMaskRCNNConfig

Frozen architecture spec. Use the mask_rcnn_resnet50_fpn factory for the COCO-pretrained configuration (num_classes = 91).

Attributes

configMaskRCNNConfig

Stored copy of the config that built this model.

backbone_BackboneWithFPN

ResNet-50 body + fpn producing five feature maps [P2, P3, P4, P5, pool] at strides 4/8/16/32/64 (reused from Faster R-CNN).

rpn_RegionProposalNetwork

Proposal head shared across all pyramid levels (reused).

roi_heads_MaskRoIHeads

box_head + box_predictor (reused) plus mask_head (MaskRCNNHeads) + mask_predictor (MaskRCNNPredictor).

Notes

See He et al., "Mask R-CNN", ICCV 2017 (arXiv:1703.06870), Ren et al., "Faster R-CNN", NeurIPS 2015, and Lin et al., "Feature Pyramid Networks for Object Detection", CVPR 2017. The model expects an already resized

normalised image batch; final per-instance detections + masks come from postprocess.

Examples

>>> import lucid
>>> from lucid.models.vision.mask_rcnn import mask_rcnn_resnet50_fpn
>>> model = mask_rcnn_resnet50_fpn().eval()
>>> x = lucid.randn(1, 3, 224, 224)
>>> out = model(x)
>>> out.logits.shape[-1]   # num_classes
91
>>> out.pred_masks.shape[-2:]
(28, 28)

Used by 2

Constructors

dunder

init

→None

__init__(config: MaskRCNNConfig)

source edit

Instance methods

forward

→InstanceSegmentationOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, proposals: list[Tensor] | None = None)

source edit

Run Mask R-CNN on a (pre-processed) image batch.

Parameters

xTensor

(B, C, H, W) resized + normalised image batch.

targetslist[dict[str, Tensor]] | None= None

Optional per-image {"boxes": (M, 4) xyxy, "labels": (M,), "masks": (M, H, W) binary}. When given, the five-term loss L_rpn_obj + L_rpn_reg + L_cls + L_box + L_mask is computed. "masks" may be omitted, in which case L_mask is zero and only the detector trains.

proposalslist[Tensor] | None= None

Optional precomputed per-image proposals. When None the RPN generates them.

Returns

InstanceSegmentationOutput

InstanceSegmentationOutput with raw RoI-head outputs: logits : (Σ proposals, num_classes) class logits. pred_boxes : (Σ proposals, num_classes, 4) per-class boxes. pred_masks : (Σ proposals, num_classes, 28, 28) mask logits. loss : scalar sum of the five terms, or None.

Raises

ValueError

If targets is given but the RPN did not run.

postprocess

→list of dict

postprocess(output: InstanceSegmentationOutput, image_sizes: list[tuple[int, int]] | None = None, proposals: list[Tensor] | None = None, features: list[Tensor] | None = None)

source edit

Box post-process → per-detection mask gather.

Mirrors the reference inference flow: run the box branch post-processing (softmax → per-class score filter → per-class NMS → top-max_detections), then sigmoid the per-RoI mask logits and gather the channel for each detection's predicted class.

Parameters

outputInstanceSegmentationOutput

Raw RoI-head outputs from forward.

image_sizeslist of (H, W)= None

Per-image extent. When given, image i's boxes are re-clipped to image_sizes[i], as in Faster R-CNN (decoding clips only to the padded batch canvas), and each detection's mask is pasted onto its clipped box in an H x W canvas and binarised at config.mask_thresh. When omitted, masks stay 2 * roi_mask_size-square (28 x 28 by default) probabilities in RoI coordinates.

proposalslist of Tensor= None

Per-image proposals the RoI features were sampled from; falls back to output.proposals, which forward fills in.

featureslist of Tensor= None

Multi-scale FPN feature maps; accepted for API symmetry with the detector-stage post-processor but not consumed here — the re-alignment on the final boxes reads output.hidden_states.

Returns

list of dict

One dict per image with "boxes" (D, 4), "scores" (D,), "labels" (D,) int64, and "masks" — the channel of each detection's predicted class, as (D, 1, 28, 28) sigmoid probabilities or, with image_sizes, as (D, 1, H, W) binary masks. An image with no detections has D = 0 and the same mask shape.

Examples

>>> import lucid
>>> from lucid.models import create_model
>>> lucid.manual_seed(0)
>>> model = create_model(
...     "mask_rcnn_resnet50_fpn", num_classes=3,
...     backbone_layers=(1, 1, 1, 1), fpn_out_channels=32,
...     roi_representation=64, mask_hidden_channels=16,
...     mask_num_convs=1, mask_predictor_hidden=16,
...     rpn_post_nms_top_n=10,
... ).eval()
>>> out = model(lucid.randn((1, 3, 64, 64)))
Without image_sizes there is one mask per box, each a 28 x 28
grid of probabilities in that box's own frame.
>>> det = model.postprocess(out)[0]
>>> det["masks"].shape[0] == det["boxes"].shape[0]
True
>>> det["masks"].shape[1:]
(1, 28, 28)
With it, each grid is stretched onto its box inside an image-sized
canvas and thresholded, so it can be laid over the picture as is.
>>> masks = model.postprocess(out, image_sizes=[(64, 64)])[0]["masks"]
>>> masks.shape[1:]
(1, 64, 64)
>>> bool(((masks == 0) | (masks == 1)).all())
True
Inside a padded batch an image's own extent is smaller than the
canvas.  For one 48 tall and 40 wide the boxes are clipped to it,
and the masks come back on its 48 x 40 frame.
>>> det = model.postprocess(out, image_sizes=[(48, 40)])[0]
>>> xs, ys = det["boxes"][:, 0::2], det["boxes"][:, 1::2]
>>> bool((xs <= 40).all()), bool((ys <= 48).all())
(True, True)
>>> det["masks"].shape[1:]
(1, 48, 40)

>>> import lucid >>> from lucid.models.vision.mask_rcnn import mask_rcnn_resnet50_fpn >>> model = mask_rcnn_resnet50_fpn().eval() >>> x = lucid.randn(1, 3, 224, 224) >>> out = model(x) >>> out.logits.shape[-1] # num_classes 91 >>> out.pred_masks.shape[-2:] (28, 28)

forward

→InstanceSegmentationOutput

forward(x: Tensor, targets: list[dict[str, Tensor]] | None = None, proposals: list[Tensor] | None = None)

source edit

Run Mask R-CNN on a (pre-processed) image batch.

Parameters

xTensor

(B, C, H, W) resized + normalised image batch.

targetslist[dict[str, Tensor]] | None= None

proposalslist[Tensor] | None= None

Optional precomputed per-image proposals. When None the RPN generates them.

Returns

InstanceSegmentationOutput

Raises

ValueError

If targets is given but the RPN did not run.

>>> import lucid >>> from lucid.models import create_model >>> lucid.manual_seed(0) >>> model = create_model( ... "mask_rcnn_resnet50_fpn", num_classes=3, ... backbone_layers=(1, 1, 1, 1), fpn_out_channels=32, ... roi_representation=64, mask_hidden_channels=16, ... mask_num_convs=1, mask_predictor_hidden=16, ... rpn_post_nms_top_n=10, ... ).eval() >>> out = model(lucid.randn((1, 3, 64, 64))) Without image_sizes there is one mask per box, each a 28 x 28 grid of probabilities in that box's own frame. >>> det = model.postprocess(out)[0] >>> det["masks"].shape[0] == det["boxes"].shape[0] True >>> det["masks"].shape[1:] (1, 28, 28) With it, each grid is stretched onto its box inside an image-sized canvas and thresholded, so it can be laid over the picture as is. >>> masks = model.postprocess(out, image_sizes=[(64, 64)])[0]["masks"] >>> masks.shape[1:] (1, 64, 64) >>> bool(((masks == 0) | (masks == 1)).all()) True Inside a padded batch an image's own extent is smaller than the canvas. For one 48 tall and 40 wide the boxes are clipped to it, and the masks come back on its 48 x 40 frame. >>> det = model.postprocess(out, image_sizes=[(48, 40)])[0] >>> xs, ys = det["boxes"][:, 0::2], det["boxes"][:, 1::2] >>> bool((xs <= 40).all()), bool((ys <= 48).all()) (True, True) >>> det["masks"].shape[1:] (1, 48, 40)