RAdam optimiser — Rectified Adam (Liu et al., 2019).
Computes a variance-based rectification factor derived from the simple moving average length . When exceeds a threshold (the paper uses ) the adaptive learning rate is well-conditioned and we apply the standard bias-corrected Adam step scaled by ; otherwise we fall back to a plain momentum step using only , which avoids the early-training divergence caused by an under-sampled second moment.
Math
If :
otherwise (fallback to plain momentum):
References
Liu, Jiang, He, Chen, Liu, Gao & Han, "On the Variance of the Adaptive Learning Rate and Beyond" (ICLR 2020, arXiv 2019).
See Also
Adam, AdamW, NAdam.
Attributes
lr_float1e-3.beta1_, beta2_, eps_, weight_decay_floatAdam.m_, v_Storage[]step_count_int64Notes
and depend only on and , so they are computed once per step outside the per-parameter loop. Weight decay (if non-zero) is added to in the standard L2 form.
Constructors
1Construct an RAdam optimiser bound to params.
Parameters
paramsvector of TensorImpl shared pointerslrfloat= None1e-3.beta1, beta2float= None0.9 / 0.999.epsfloat= None1e-8.weight_decayfloat= None0.Virtual methods
2Methods
3Allocate zero-initialised and buffers for this slot.
Schema identifier for state_dict serialisation.
Returns
str"radam_v1".
Compute and ; apply the rectified adaptive update when , else fall back to a plain momentum update.