Reference: Bruce, Jake, et al., "Genie: Generative Interactive Environments", ICML 2024 (arXiv:2402.15391) — Table 7 for the tokenizer, Table 5 for the latent action model, Table 12 for the dynamics model.

The paper reports 10.7B parameters: a 200M tokenizer, a 300M latent action model and a 10.1B dynamics model. This build has 20.2B — 388M, 675M and 19.1B — and no choice of the values the paper leaves out reconciles its tables with those counts. The latent action model's two attention layers per block come to 339M at the tabulated widths, over its 300M before any feed-forward layer; the tokenizer would need a feed-forward a tenth of its width and the dynamics model two-fifths of its, which disagree with each other and with every transformer at this scale. The tables are what is built. At float32 that is about 81 GB; the scaling study's smaller dynamics models (Table 10) are overrides of the dynamics_* sizes.