ViT-Base/16 backbone (Dosovitskiy et al., 2020).
Builds the canonical ViT-Base configuration with patch size 16:
dim=768, depth=12, num_heads=12, mlp_ratio=4.0. With
the default 224x224 input this yields patch
tokens plus one CLS token, for an input sequence length of 197 to
the transformer encoder. Approximately 86M parameters.
Model Size
Parameters
pretrainedbool= FalseIf
True, loads ImageNet-21k pretrained weights when available
in the model zoo. Defaults to False (random initialization).**overridesobject= {}Keyword overrides applied on top of the canonical ViT-Base/16
config — e.g.
image_size=384, in_channels=1, or
dropout=0.1. Each override must match a field of
ViTConfig.Returns
ViTA ViT backbone instance returning a flat (B, 768) CLS
feature.
Notes
ViT-Base/16 was pretrained on JFT-300M / ImageNet-21k and reported 84.15% top-1 / 97.20% top-5 on ImageNet-1k after fine-tuning at 384x384 in Dosovitskiy et al. (2020, Table 5). See arXiv:2010.11929.
Examples
>>> import lucid
>>> from lucid.models.vision.vit import vit_base_16
>>> model = vit_base_16()
>>> x = lucid.randn(1, 3, 224, 224)
>>> feat = model.forward_features(x)
>>> feat.shape
(1, 768)