V-JEPA

V-JEPA learns from video by predicting features of what it cannot see. A clip of $T$ frames becomes $T/2 \times H/16 \times W/16$ tubelet tokens; a mask hides most of them; the encoder $f_\theta$ reads what is left, and a predictor $g_\phi$ must produce the output of an exponentially averaged encoder $f_{\bar\theta}$ at the hidden positions:

Two mask collections, not one. Eight blocks covering 15% of the frame each, and two covering 70%, both running the full depth of the clip in time. Their union hides about 90% of the tokens. A mask that leaves a tube visible through time makes the task trivial --- the answer is in the next frame --- so both collections extend through it, and the model is asked the two questions at once, with a learned mask token of its own for each.

The target encoder sees everything; the context encoder does not. Masking here removes tokens rather than attending around them, so the context encoder's sequence is genuinely shorter, while the averaged encoder reads the whole clip and its output is normalised before any block is taken from it. No gradient flows into it at all.

Prediction in feature space is what makes the video tractable. A generative video objective must settle every detail that differs between plausible continuations; this one may drop whatever the averaged encoder has already learned to drop. Frozen, with an attentive probe on top, the result reaches 82.0% on Kinetics-400 and 71.4% on Something-Something-v2 --- the latter being the benchmark that punishes models which only recognise appearance.