PlaNet

PlaNet learns the dynamics of an environment in latent space and plans there, never reconstructing a frame in order to choose an action. The difficulty it addresses is that pixels are a partial observation: the agent must both remember what it has seen and stay uncertain about what it has not.

Its answer is the recurrent state-space model, which refuses to pick between a deterministic and a stochastic latent and carries both. A deterministic path $h_t$ is updated by a gated recurrence, and a stochastic latent $s_t$ is drawn conditioned on it:

The motivation is stated as an ablation in the paper: a purely stochastic model cannot retain information over many steps, because each step's sampling noise degrades it; a purely deterministic model cannot represent the several futures a partially-observed environment permits, so it averages them and blurs. Only the model carrying both matches the reported performance.

Training maximises a variational bound. An encoder supplies an approximate posterior $q(s_t \mid h_t, o_t)$ that has seen the frame; the objective rewards reconstructing the observation and the reward from that posterior while pulling the dynamics' own prior toward it:

The KL is clamped below at a free-nats threshold, so no gradient is spent driving an already-small divergence lower — without it the posterior collapses onto the prior and the latent stops carrying the observation. The asymmetry is worth noting: the reconstruction term never reaches the prior head at all, since nothing reconstructed is computed from the prior. The KL is the prior's only teacher.

Once trained, planning runs the prior forward alone — the model imagines action sequences and scores them without touching the environment. That the rollout happens entirely in a compact latent makes searching over thousands of candidate trajectories affordable, which is what lets PlaNet match model-free agents at a fraction of the episodes.