Stable Diffusion

Diffusion in pixel space spends most of its capacity on detail the eye discards. Latent diffusion splits the problem in two: an autoencoder learns a perceptually equivalent space at a fraction of the resolution, and the diffusion model runs entirely inside it.

The first stage is an encoder $\mathcal{E}$ and decoder $\mathcal{D}$ with a downsampling factor $f = H/h = W/w$. At $f = 8$ a $512 \times 512 \times 3$ image becomes a $64 \times 64 \times 4$ latent — 48 times fewer values, so every subsequent convolution and attention costs proportionally less. The paper regularises this space rather than leaving it free: a slight KL penalty toward $\mathcal{N}(0, I)$ keeps the latents from drifting to arbitrary scale, which matters because the diffusion process assumes unit-ish variance.

The second stage is a time-conditional U-Net trained on the usual denoising objective, but on latents:

Conditioning enters by cross-attention rather than concatenation, which is what makes the modality of $y$ irrelevant to the U-Net. With $\varphi_i(z_t)$ the flattened intermediate representation,

so the latent asks the questions and the conditioning answers them. A text encoder, a class embedding and a layout map all present the same interface — a sequence of vectors — and the U-Net never learns which it is reading.

$\tau_\theta$ is deliberately left open by the paper, which evaluates several. The released models use a frozen CLIP text encoder, so the width below is CLIP's and the conditioning sequence is its 77 tokens.