class

RoFormerMLMWeights

extendsWeightsEnum

RoFormerMLMWeights()

Pretrained weight tags for lucid.models.roformer_mlm.

RoFormer encoder + tied MLM head over the 50 000-token Chinese vocab.

CLUECORPUSSMALLWeightEntry

CLUECorpusSmall Chinese pre-training checkpoint (encoder + MLM head) sourced from transformers/junnyu/roformer_chinese_base.

DEFAULTWeightEntry

Alias for CLUECORPUSSMALL.

Reference: Su, Lu, Pan, Murtadha, Wen, Liu, "RoFormer: Enhanced Transformer with Rotary Position Embedding", 2021 (arXiv:2104.09864).

>>> from lucid.models import roformer_mlm
>>> model = roformer_mlm(pretrained=True).eval()