SENet

SENet introduces the Squeeze-and-Excitation block, a tiny drop-in channel-attention module that recalibrates the relative importance of each feature-map channel after every convolutional stage. The block runs in three phases.

The squeeze phase reduces each channel to a single scalar by global average pooling over the spatial dimensions:

$$z_c = \frac{1}{H \cdot W} \sum_{i=1}^{H} \sum_{j=1}^{W} u_c(i, j),$$

yielding a channel descriptor $z \in \mathbb{R}^{C}$ that summarises the global response of every channel.

The excitation phase passes $z$ through a small two-layer fully-connected bottleneck with a reduction ratio $r$ (typically 16) and a sigmoid output:

$$s = \sigma\big( W_2 \,\delta( W_1 z ) \big), \qquad W_1 \in \mathbb{R}^{C/r \times C}, \; W_2 \in \mathbb{R}^{C \times C/r},$$

where $\delta$ is ReLU and $\sigma$ is the sigmoid. The output $s \in (0, 1)^{C}$ is interpreted as per-channel gating weights.

Finally the scale phase modulates the original feature map channel-wise:

$$\tilde{u}_c = s_c \cdot u_c.$$

Plugging this block into the end of every residual unit of a ResNet-50 (giving SE-ResNet-50) adds only ~10% extra parameters and negligible FLOPs while reducing ImageNet top-5 error by roughly 1.5 percentage points — a result that won the ILSVRC 2017 classification challenge. In Lucid SENetConfig parameterises every SE-ResNet variant (18 / 34 / 50 / 101 / 152) with the same layers and block_type knobs as ResNet plus the SE-specific reduction ratio.