U-Net

U-Net is a symmetric encoder–decoder architecture with dense skip connections, designed for biomedical segmentation where training data is scarce and every pixel matters.

The encoder ("contracting path") follows the FCN philosophy of a classification CNN: pairs of $3 \times 3$ convolutions (with ReLU) followed by $2 \times 2$ max-pooling, doubling channel depth at every spatial halving:

The decoder ("expansive path") mirrors this structure: each stage performs a $2 \times 2$ up-convolution (transposed conv) that halves the channel count and doubles spatial size, concatenates the corresponding encoder feature map along the channel axis, and applies two $3 \times 3$ convs:

The crucial design choice is the skip connection by concatenation (not addition): it forwards the encoder's high-resolution but semantically shallow features directly into the decoder, letting the convs combine them with the deeper, semantically rich up-sampled features. This is what allows the network to produce crisp pixel-level boundaries with very few training images (the paper trains on 30 cell images via heavy elastic augmentation).

The final layer is a $1 \times 1$ convolution mapping to $K$ classes (softmax + per-pixel cross-entropy). Weighted-pixel loss is recommended in the paper to emphasise thin separating boundaries between touching cells — a small but practically important detail for cell segmentation. U-Net's skip-connection pattern has since become the default starting point for almost every dense-prediction task, from natural images to volumetric medical scans (3-D U-Net, V-Net, …).