Genie

Genie turns unlabelled video into a playable environment. A user gives it one image and presses one of $|A| = 8$ buttons; it answers with the next frame, and keeps answering. Nothing in its training data says what the buttons are --- the meaning of each is learned, and emerges as left, right, jump and no-op on the platformer games it was trained on.

Three models, one block. Each component is a spatiotemporal (ST) transformer over a video of $T$ frames, each cut into $N$ patches. A block attends spatially over the $N$ tokens of one frame, then temporally over the $T$ tokens at one position under a causal mask, and only then applies a single feed-forward layer:

The cost is $O(TN^2 + NT^2)$ rather than $O(T^2N^2)$, which is what makes 16 frames of 920 tokens trainable at all.

The latent action model reads frames $x_{1:t+1}$ and emits a continuous vector for the transition $t \to t+1$, which is quantised against a codebook of eight entries. A decoder must then rebuild $x_{t+1}$ from the past frames and that code alone, so the code has to carry exactly what the history does not already predict --- the choice. Eight codes are too few to smuggle the next frame through, which is the whole point.

The dynamics model is a MaskGIT transformer over the tokens of a separately trained VQ video tokenizer. The paper writes its objective over the whole next frame,

with the input tokens masked at a rate drawn from $U[0.5, 1]$ and the embedding of latent action $\tilde a_t$ added rather than concatenated. Read literally that predicts frame $t$ from the frames before it alone, which cannot be refined: the 25 MaskGIT steps a frame is generated over need the tokens revealed so far to re-enter the model. So the frame is filled in in place here --- frame $t$'s masked tokens, conditioned on $z_{<t}$, on what is already revealed of $z_t$, and on $\tilde a_{t-1}$ --- and the loss covers the masked positions, as MaskGIT's does.

At play time a frame starts fully masked and is revealed over those steps, keeping the most confident samples each time. Only the latent action model's codebook is needed then: a user's button is an index into it. The paper reports 10.7B parameters --- a 200M tokenizer, a 300M latent action model and a 10.1B dynamics model.