LeNet

LeNet-5 is the prototypical convolutional neural network and the direct ancestor of every modern vision architecture. Designed for handwritten-digit recognition on 32×32 grayscale inputs, it interleaves learned $5\times5$ convolutions with fixed $2\times2$ sub-sampling (average pooling) layers, ending in two fully-connected layers and a Gaussian-RBF output layer.

The key insight of LeCun et al. was that local receptive fields, shared weights, and spatial sub-sampling — together — give a model that is approximately invariant to small translations, scalings, and distortions of the input. Convolution enforces weight sharing: the same $5\times5$ filter slides across the image, so a feature detector learned in one location applies everywhere. Sub-sampling layers then pool responses over a neighbourhood, yielding coarser feature maps that are robust to the exact position of each feature.

Formally, a LeNet layer computes

where $\phi$ is $\tanh$ in the original paper (or ReLU in modern reimplementations) and the kernel $W^{(k)}$ is shared across all spatial positions $(i, j)$. Despite its small size by today's standards (≈60 k parameters), LeNet established the convolutional template — conv → nonlinearity → pool → repeat → flatten → MLP — that is still recognisable in every CNN that followed.