DiT

A diffusion model needs a network that maps a noised latent and a timestep to a prediction of the noise. Nothing about that job requires convolutions, and DiT tests the claim directly: patchify the latent into a sequence, run a standard Vision Transformer over it, and decode back.

The decoder emits twice the input channels because the model predicts a noise and a diagonal covariance, following ADM's parameterisation of $\Sigma_\theta$.

Where the conditioning enters is the finding. The paper compares four ways of admitting the timestep and class label and they are not close. In-context conditioning appends them as tokens; cross attention gives them their own attention layer at about 15% more compute; adaptive layer norm regresses a per-channel shift and scale from their sum; and adaLN-Zero adds a third regressed vector, a gate $\alpha$ applied immediately before each residual add:

Initialising the projection that produces $\alpha$ at zero makes every block the identity at step zero. That single change roughly halves the FID of the in-context variant at 400K steps, and it is the cheapest of the four in Gflops.

Scale is the other finding. Across twelve models — four widths crossed with three patch sizes — FID correlates with the backbone's Gflops at $-0.93$, and parameter count is not the predictor: holding parameters fixed while shrinking the patch improves FID substantially, because more tokens mean more compute. Sampling compute does not substitute for it; a small model given eight times the sampling steps still loses to a large one.