Adagrad optimiser — accumulates squared gradients without forgetting.
Maintains a running sum of per parameter and scales the learning rate by its inverse square root. The accumulator grows monotonically so the effective learning rate decays to zero in the limit — well suited to sparse features (NLP, embeddings) but typically too aggressive for dense deep-network weights.
Math
References
Duchi, Hazan & Singer, "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization" (JMLR 12, 2011).
See Also
Adadelta, RMSprop — both add a forgetting factor
to address Adagrad's monotonic learning-rate decay.
Parameters
paramsvector of TensorImpllrfloat, default 1e-2epsfloat, default 1e-10weight_decayfloat, default 0.0initial_accumulator_valuefloat, default 0.0sum_sq_grad_. A non-zero value avoids dividing by a near-zero accumulator on the very first step.Attributes
sum_sq_grad_vector of StorageNotes
Because the accumulator never decays, the effective per-parameter learning rate is non-increasing — once a coordinate has seen large gradients its step shrinks for the rest of training. This is precisely what makes Adagrad strong on sparse problems and weak on long dense-network runs.
Constructors
1Construct the optimiser and resize per-slot state vectors.
Parameters
paramsvector of TensorImpllrfloat, default 1e-2epsfloat, default 1e-10weight_decayfloat, default 0.0initial_accumulator_valuefloat, default 0.0sum_sq_grad_.Virtual methods
2Methods
3Allocate the squared-gradient accumulator for parameter i.
Parameters
iintNotes
When initial_accumulator_value_ != 0 the buffer is filled
with that scalar rather than zeros.
Return the versioned state-dict identifier.
Returns
str"adagrad_v1".
Apply one Adagrad update to parameter i.
Parameters
iintgStoragep on this step.Notes
Accumulates into sum_sq_grad_[i] then applies
.