optim

Stochastic Gradient Descent optimizer with optional momentum and weight decay.

Adaptive Moment Estimation optimizer (Kingma & Ba, 2015).

Adam optimizer with decoupled weight decay regularisation.

Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) optimizer.

Root Mean Square Propagation optimizer.

Adaptive Gradient optimizer.

Adadelta optimizer — an adaptive learning rate method with no global LR.

Adamax optimizer — a variant of Adam based on the infinity norm.

Rectified Adam optimizer with variance-adapted step size.

Nesterov-accelerated Adaptive Moment Estimation optimizer.

Averaged Stochastic Gradient Descent optimizer.

Resilient Backpropagation optimizer.

Adam optimizer designed for sparse gradient workloads.

Decay the learning rate by a fixed multiplicative factor every fixed number of epochs.

Decay the learning rate by a constant factor every epoch.

Decay the learning rate by a fixed factor at a list of epoch milestones.

Anneal the learning rate following a cosine curve over `T_max` epochs.

Set the learning rate using a user-supplied multiplicative factor function.

Cycle the learning rate between `base_lr` and `max_lr`.

Reduce the learning rate when a monitored metric stops improving.

Noam learning rate schedule from the original Transformer paper.

Multiply the learning rate by a factor returned by a function each epoch.

Linearly interpolate the learning rate from a start factor to an end factor.

Hold the learning rate at a constant scaled value, then restore the base LR.

Decay the learning rate using a polynomial function over a fixed number of steps.

Cosine annealing with periodic warm restarts (SGDR).

1Cycle learning rate policy.

Chain multiple schedulers and switch between them at epoch milestones.

Apply multiple schedulers simultaneously, compounding their LR changes.