ViT-Huge/14 backbone (Dosovitskiy et al., 2020).
The canonical ViT-Huge configuration with patch size 14:
dim=1280, depth=32, num_heads=16, mlp_ratio=4.0. With
a 224x224 input it produces patch tokens
(+1 CLS = 257 total) feeding 32 transformer blocks of width 1280.
Approximately 632M parameters — the largest variant in the
paper.
Model Size
Parameters
pretrainedbool= FalseIf
True, loads JFT-300M / ImageNet-21k pretrained weights when
available. Defaults to False.**overridesobject= {}Keyword overrides on top of the canonical ViT-Huge/14 config.
Returns
ViTA ViT backbone returning a flat (B, 1280) CLS feature.
Notes
ViT-Huge/14 achieves 88.55% top-1 on ImageNet-1k when fine-tuned at 518x518 after JFT-300M pretraining (Dosovitskiy et al., 2020, Table 2) — the state-of-the-art result reported in the paper at release. Training from scratch on ImageNet-1k alone is impractical at this scale; pretrained weights are strongly recommended.
Examples
>>> import lucid
>>> from lucid.models.vision.vit import vit_huge_14
>>> model = vit_huge_14()
>>> x = lucid.randn(1, 3, 224, 224)
>>> model.forward_features(x).shape
(1, 1280)