Dreamer

Dreamer inherits PlaNet's recurrent state-space model and throws away its planner. Where PlaNet searches over action sequences at every step — thousands of imagined trajectories to choose one action — Dreamer learns a policy, so acting costs a single forward pass.

The trade is that a learned policy needs a learning signal, and the paper's contribution is where that signal comes from. An actor $a_\tau \sim q_\phi(a_\tau \mid s_\tau)$ and a critic $v_\psi(s_\tau)$ are trained entirely on trajectories the world model imagines, never on environment interaction. Because the imagined states are produced by a differentiable transition and the actions are reparameterised samples, the gradient of a predicted return flows backwards through the dynamics into the actor:

This is what separates it from a model-free actor-critic, which can only estimate that gradient from scalar rewards it has actually observed. Here the model supplies an analytic path.

The target $V_\lambda$ is an exponentially-weighted average of $k$-step returns — TD($\lambda$) computed inside the imagination:

with each $V_N^k$ summing discounted imagined rewards and bootstrapping from the critic. The weighting is the usual bias-variance dial: short horizons trust the critic, long ones trust the model, and $\lambda$ interpolates. The critic then regresses onto that same quantity, which is why the two networks are trained together but with the target held fixed.

Imagination is short — 15 steps in the paper — because the model's error compounds, and the critic's bootstrap is what lets a short horizon still represent long-term value.