DenseNet

DenseNet takes the residual-shortcut idea to its logical extreme. Inside each dense block, every layer receives as input the concatenated feature maps of all previous layers in the block, not just the immediately preceding layer. For a block of $L$ layers the $\ell$-th layer computes

where $[\cdot]$ denotes channel-wise concatenation and $H_\ell$ is a BN → ReLU → $3\times3$ Conv composite (optionally preceded by a $1\times1$ bottleneck). Each $H_\ell$ contributes a small fixed number $k$ of new feature maps — the growth rate — so the input width of layer $\ell$ grows linearly as $k_0 + (\ell - 1)\,k$.

This dense connectivity has three notable consequences. First, every layer has direct supervision-style access to the loss gradient through the concatenation paths, easing optimisation in very deep networks. Second, parameter efficiency is excellent: because each layer only adds $k$ (typically 12, 24, or 32) new channels rather than transforming the full activation, a DenseNet with 0.8 M parameters matches the accuracy of a 1.7 M parameter ResNet on CIFAR. Third, feature reuse is explicit — later layers can directly combine raw early features with refined deep features without re-encoding them.

The four ImageNet variants — DenseNet-121, 169, 201, and 264 — differ only in the per-block layer counts (block_config). All use four dense blocks separated by transition layers ($1\times1$ Conv + $2\times2$ average pool) that compress the channel count and halve the spatial resolution. DenseNet-121 reaches a top-1 ImageNet error of 25.0% with 8 M parameters, substantially better parameter efficiency than the contemporary ResNet-50.