CosineAnnealingLR

Cosine-annealing schedule from $\eta_{\max}$ down to $\eta_{\min}$ over $T_{\max}$ epochs, following half a cosine wave.

Smoothly anneals the lr so training spends the early epochs at high lr (exploration) and the late epochs at low lr (convergence). Often combined with warm restarts (SGDR) for cyclic re-exploration.

Math

for $t \in [0, T_{\max}]$; saturates at $\eta_{\min}$ for $t > T_{\max}$.

References

Loshchilov & Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts" (ICLR 2017) — Lucid does not yet implement the warm-restart variant.