data
GenieOutput
extends
ModelOutputGenieOutput(loss: Tensor, tokenizer_loss: Tensor, latent_action_loss: Tensor, dynamics_loss: Tensor, reconstruction: Tensor, prediction: Tensor, tokens: Tensor, actions: Tensor, logits: Tensor)The three training objectives, and what each was computed from.
Attributes
lossTensorSum of the three objectives. Scalar. Each reaches only its own
network, so one
backward trains all three independently.tokenizer_lossTensorReconstruction error of the video tokenizer plus its codebook and
commitment terms.
latent_action_lossTensorError predicting frames
2..T from the frames before each and a
latent action, plus the action codebook terms.dynamics_lossTensorCross-entropy over the masked tokens of frames
2..T.reconstructionTensorThe tokenizer's reconstruction,
(B, T, C, H, W).predictionTensorThe latent action model's predicted frames,
(B, T - 1, C, H, W).tokensTensorThe tokenizer's codes,
(B, T, N).actionsTensorThe latent action of each transition,
(B, T - 1).logitsTensorThe dynamics model's logits,
(B, T, N, num_codes).Examples
>>> import lucid
>>> from lucid.models.generative.genie import GenieOutput
>>> zero = lucid.zeros(())
>>> out = GenieOutput(
... loss=zero, tokenizer_loss=zero, latent_action_loss=zero,
... dynamics_loss=zero, reconstruction=lucid.zeros(1, 4, 3, 8, 8),
... prediction=lucid.zeros(1, 3, 3, 8, 8),
... tokens=lucid.zeros(1, 4, 4).to(lucid.int64),
... actions=lucid.zeros(1, 3).to(lucid.int64),
... logits=lucid.zeros(1, 4, 4, 16))
>>> out.actions.shape
(1, 3)