Adadelta optimiser — Adagrad without a fixed learning rate.
Replaces Adagrad's monotonically growing accumulator with an EMA of
squared gradients and additionally maintains an EMA of
squared parameter updates . The ratio of the
two acts as a self-tuning step size, so lr only serves as a
global scale and is conventionally left at 1.0.
Math
References
Zeiler, "ADADELTA: An Adaptive Learning Rate Method" (arXiv:1212.5701, 2012).
See Also
Adagrad, RMSprop
Parameters
paramsvector of TensorImpllrfloat, default 1.0rhofloat, default 0.9epsfloat, default 1e-6weight_decayfloat, default 0.0Attributes
sq_avg_vector of Storageaccumulated_update_vector of StorageNotes
Adadelta's unit-matching argument: the ratio has the same units as (rather than ), which is the original motivation for the method.
Constructors
1Construct the optimiser and resize per-slot state vectors.
Parameters
paramsvector of TensorImpllrfloat, default 1.0rhofloat, default 0.9epsfloat, default 1e-6weight_decayfloat, default 0.0Virtual methods
2Methods
3Allocate the two per-slot EMA buffers for parameter i.
Parameters
iintNotes
Both sq_avg_[i] and accumulated_update_[i] are
zero-initialised.
Return the versioned state-dict identifier.
Returns
str"adadelta_v1".
Apply one Adadelta update to parameter i.
Parameters
iintgStoragep on this step.Notes
Both running averages are advanced and the unit-matched step is applied in a single fused pass.