NICE

NICE is the first normalizing flow trained on images: instead of bounding the likelihood (VAE) or factorising it over pixels (autoregressive models), it learns an invertible map $h = f(x)$ onto a latent space with a factorised prior, so the exact data log-density falls straight out of the change-of-variables formula

$$\log p_X(x) = \log p_H\big(f(x)\big) + \log\Big|\det \frac{\partial f}{\partial x}\Big|.$$

Nothing is approximated — no variational gap, no Markov chain — but every layer now has to be invertible and have a cheap Jacobian determinant, which is a severe architectural constraint.

The construction that satisfies both is the additive coupling layer. Partition the dimensions into two blocks $(x_{I_1}, x_{I_2})$ and update only the second, conditioned on the first:

$$\begin{aligned} y_{I_1} &= x_{I_1}, \\ y_{I_2} &= x_{I_2} + m(x_{I_1}). \end{aligned}$$

The Jacobian is unit lower-triangular, so its determinant is $1$ no matter how deep the coupling network $m$ is, and the inverse is the same graph with one sign flipped, $x_{I_2} = y_{I_2} - m(y_{I_1})$. Since one block passes through untouched, consecutive layers must exchange the roles of the two blocks; at least three couplings are required before every dimension can influence every other, and the paper stacks four with deep rectified networks (linear output units) as the $m$'s.

A stack of volume-preserving maps is itself volume-preserving, so NICE closes the flow with a learned diagonal scaling $h = \exp(s) \odot h^{(4)}$ — the only stage that moves the determinant:

$$\log p_X(x) = \sum_{i=1}^{D} \Big[\log p_{H_i}\big(f_i(x)\big) + s_i\Big].$$

The scales $\exp(s_i)$ behave as learned per-dimension importance weights: directions the model captures well are given small scale, making $s$ the non-linear analogue of an eigenvalue spectrum in PCA. The prior is a standard logistic (heavier tails, used for MNIST / SVHN / CIFAR-10) or a standard normal (used for TFD).

Generation is exact and parallel — draw $h \sim p_H$, return $f^{-1}(h)$ — with no decoder and no iterative refinement. The coupling layer introduced here is the direct ancestor of RealNVP (affine couplings, multi-scale spatial masks) and Glow (invertible 1x1 convolutions); the samples look blurry by modern standards, but the recipe defined how exact-likelihood generative models are built.