Dreamer-v3

DreamerV3's contribution is that its hyperparameters stop mattering. Its predecessors needed a different divergence scale for Atari than for continuous control, a different entropy bonus, a different discount; this one setting spans them, and the paper's headline result — collecting diamonds in Minecraft from scratch — is run with the same numbers as everything else.

Four changes buy that, and each removes a scale the practitioner was otherwise forced to supply.

Symlog. Rewards differ by orders of magnitude between domains, and a network fitted to one range is wrong on another. Compressing them through $\mathrm{sign}(x)\ln(|x| + 1)$ leaves small values alone and folds large ones in, so the same head covers both.

Two-hot regression. Reward and value heads predict a distribution over exponentially spaced bins rather than a number. A squared error's gradient scales with the error, which is precisely the coupling to reward magnitude the paper is trying to break; a cross-entropy's does not. It also lets a prediction be bimodal, which the mean of a squared-error head cannot represent.

Free bits. The divergence is clipped below at one nat per variable:

$$\mathcal{L}_{\mathrm{dyn}} = \max\big(1,\; \mathrm{KL}[\mathrm{sg}(q) \,\|\, p]\big), \qquad \mathcal{L}_{\mathrm{rep}} = \max\big(1,\; \mathrm{KL}[q \,\|\, \mathrm{sg}(p)\big]),$$

weighted $\beta_{\mathrm{dyn}} = 1$ and $\beta_{\mathrm{rep}} = 0.1$. Below one nat the term is flat, so no gradient is spent closing a gap that is already small — which is what stopped the posterior collapsing and made the KL scale a tuning knob in the first place. The categoricals are also mixed with 1% uniform, so no class can reach probability zero and strand its gradient.

Normalised returns. The actor divides its objective by the spread of its own recent returns,

$$S = \mathrm{EMA}\big(\mathrm{Per}(R^\lambda, 95) - \mathrm{Per}(R^\lambda, 5),\; 0.99\big),$$

dividing by $\max(1, S)$ so that small returns are left alone rather than amplified into noise. A fixed entropy bonus then means the same thing in every domain, because the thing it is traded against has been made scale-free.