Noam schedule — the Transformer "warm-up then decay" rule (Vaswani et al., 2017).
Linearly ramps the lr during the first warmup_steps then decays
it as the inverse square root of the step count. Scaled by
so that the peak lr matches the convention
used in the original Transformer paper.
Math
where is warmup_steps, is factor, and is the
step count (compute_lr_at uses epoch directly as ).
References
Vaswani et al., "Attention Is All You Need" §5.3 (NeurIPS 2017).
Attributes
model_size_intTransformer hidden width .
warmup_steps_intNumber of linear-ramp steps before the inverse-sqrt decay kicks in.
factor_floatGlobal scale on the schedule.
Notes
Despite the LRScheduler base, Noam is conceptually a step
schedule rather than an epoch schedule — call step once
per optimiser iteration, not once per epoch.
Constructors
1ctor
NoamScheduler
void NoamScheduler(Optimizer & opt, int model_size, int warmup_steps, double factor)Build a Noam (Transformer) schedule.
Parameters
optOptimizerTarget optimiser.
model_sizeintTransformer hidden width .
warmup_stepsintLinear-ramp length before the inverse-sqrt decay starts.
factorfloat, default 1.0Global scale on the schedule.