Pretrained weight tags for lucid.models.roformer.
Su et al. 2021 12-layer rotary encoder trunk (H=768, 123.6 M params).
Attributes
CLUECORPUSSMALLWeightEntryCLUECorpusSmall Chinese pre-training checkpoint sourced from
transformers/junnyu/roformer_chinese_base.DEFAULTWeightEntryAlias for
CLUECORPUSSMALL.Notes
Reference: Su, Lu, Pan, Murtadha, Wen, Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021 (arXiv:2104.09864).
Examples
>>> from lucid.models import roformer
>>> model = roformer(pretrained=True).eval()