V-JEPA 2

V-JEPA 2 learns a video representation by predicting the target encoder's latent features at hidden tubelet positions. A 3-D patch embedding turns a clip into a sequence of $2\times16\times16$ tubelets. The context encoder sees only the unmasked indices, while the predictor inserts learned mask tokens and reconstructs the target encoder's representation at the held-out indices. The target encoder is an exponential moving average of the context encoder and is not differentiated through.

The released backbone uses a pre-normalised transformer with three-axis rotary position encoding. Each attention head gives one third of its width to time, row and column — rounded down to an even number, so a head of 64 channels rotates 20 per axis and leaves the last four untouched. Because position enters through that rotation rather than a learned table, a clip of a different length or resolution needs no interpolation and no new parameters: the 384-pixel variant is the 256-pixel network read over a wider grid.

Two details of the released rotation are reproduced rather than corrected. The frequency half is repeated as a block before adjacent feature pairs are rotated, which pairs each rotation with a frequency that a straightforward implementation would not choose; the upstream source marks it as a bug and keeps it, because the published weights were trained through it. Fixing it yields a different model, not a better-implemented one.

The predictor is narrower than the video encoder — 384 channels and 12 heads in every released checkpoint, whatever the encoder's width — and owns ten mask-token slots, one per block the mask generator samples. It sees the context tokens and the mask tokens together, sorted into token order so that a position's identity reaches attention through the same rotary coordinates the encoder used, then answers only at the held-out positions. Targets come from the momentum encoder and are layer-normalised, so the objective compares directions in representation space rather than magnitudes.