CyclicLR

Triangular cyclic learning-rate schedule (Smith, 2015).

Oscillates between base_lr and max_lr over cycles of length total_size_ = step_size_up + step_size_down. Each cycle linearly climbs to max_lr over step_size_up epochs then descends to base_lr over step_size_down epochs. Three amplitude profiles are supported via Mode.

Math

Let $c = t / T$ be the cycle index, $\phi$ the within-cycle phase, and $A_c$ the cycle amplitude. Then

where Triangular uses $A_c = (\eta_{\max} - \eta_{\min})$ for every cycle, Triangular2 halves it as $A_c = (\eta_{\max} - \eta_{\min})/2^c$, and ExpRange decays it as $(\eta_{\max} - \eta_{\min})\,\gamma^t$.

References

Smith, "Cyclical Learning Rates for Training Neural Networks" (WACV 2017, arXiv:1506.01186).