class

ASGD

extendsOptimizer

ASGD

source edit

Averaged Stochastic Gradient Descent (Polyak-Ruppert averaging).

Runs a standard SGD-with-momentum trajectory while also maintaining a running average $\bar\theta$ of the parameter values. Once the step counter passes the warm-up threshold t0_, every step blends the current parameter into the running average using a decaying mixing coefficient $\eta_{\mathrm{avg}}$ :

\eta_{\mathrm{avg}} = \frac{1}{\alpha\, t + 1}, \qquad \bar\theta_{t+1} = (1 - \eta_{\mathrm{avg}})\, \bar\theta_t + \eta_{\mathrm{avg}}\, \theta_{t+1} - \lambda\, \bar\theta_t.

Before t0_ the average simply tracks $\theta$ ( $\bar\theta = \theta$ ).

The averaged weights $\bar\theta$ generally yield lower-variance estimators than the instantaneous parameters, especially for convex or near-convex problems, and are typically used at inference time after training has finished.

Math

Per-parameter update (post warm-up):

g_t \leftarrow g_t + \lambda\, \theta_t,

v_{t+1} = \mu\, v_t + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1},

\bar\theta_{t+1} = (1 - \eta_{\mathrm{avg}})\, \bar\theta_t + \eta_{\mathrm{avg}}\, \theta_{t+1} - \lambda\, \bar\theta_t.

References

Polyak and Juditsky, "Acceleration of stochastic approximation by averaging" (SIAM J. Control Optim., 1992).

See Also

SGD : the underlying instantaneous update.

Attributes

lr_double

Learning rate

\eta

momentum_double

Momentum coefficient

\mu

. Zero disables velocity allocation.

weight_decay_double

L2 penalty coefficient

\lambda

alpha_double

Decay exponent controlling the averaging schedule (

\eta_{\mathrm{avg}} = 1 / (\alpha t + 1)

t0_double

Number of warm-up steps before averaging starts. Stored as double to mirror the reference framework's API but compared against integer step counts.

lambd_double

Average-side decay coefficient applied to

\bar\theta

each averaging step.

moment_std::vector<Storage>

Per-parameter SGD velocity buffers (active when momentum_ != 0).

ax_std::vector<Storage>

Per-parameter running averages of the parameter trajectory. Initialised to a copy of the parameter on the first observed gradient.

step_std::vector<std::int64_t>

Per-parameter step counters driving

\eta_{\mathrm{avg}}

Notes

The step counter is maintained per parameter slot rather than as a single global counter so that parameters introduced into training late (or temporarily frozen) still see a clean averaging schedule.

Constructors

ctor

ASGD

void ASGD(int)

source edit

Construct an ASGD optimizer.

Parameters

paramsstd::vector<std::shared_ptr<TensorImpl>>

Parameters to optimise.

lrdouble= None

Learning rate

\eta

(default 1e-3).

momentumdouble= None

SGD momentum coefficient

\mu

(default 0.0).

weight_decaydouble= None

L2 regularisation coefficient

\lambda

(default 0.0).

alphadouble= None

Decay exponent for the averaging schedule (default 0.75).

t0double= None

Warm-up step count before averaging engages (default 1e6).

lambddouble= None

Average-side decay coefficient (default 1e-4).

Virtual methods

virtconst

lr

double lr()

source edit

Current learning rate $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Update the learning rate from a scheduler.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate per-slot state on the first observed gradient.

Materialises the velocity buffer (only when momentum_ != 0) and the running-average buffer ax_ initialised to a copy of the current parameter. The per-slot step counter is reset to 0.

Parameters

istd::size_t

Slot index.

pconst std::shared_ptr<TensorImpl>&

Parameter whose layout dictates the state buffer shapes.

fnconst

state_dict_id

int state_dict_id()

source edit

Checkpoint identifier ("asgd_v1").

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply the ASGD update for one parameter slot.

Performs the standard SGD-with-momentum step on $\theta$ , then — once the per-slot step counter has passed t0_ — updates the running average $\bar\theta$ using a $1/(\alpha t + 1)$ schedule attenuated by the average-side decay $\lambda$ .

Parameters

istd::size_t

Slot index into params_, moment_, ax_, step_.

pstd::shared_ptr<TensorImpl>&

Parameter to update in place.

gconst Storage&

Accumulated gradient for this step.

class

ASGD

extendsOptimizer

ASGD

source edit

Averaged Stochastic Gradient Descent (Polyak-Ruppert averaging).

\eta_{\mathrm{avg}} = \frac{1}{\alpha\, t + 1}, \qquad \bar\theta_{t+1} = (1 - \eta_{\mathrm{avg}})\, \bar\theta_t + \eta_{\mathrm{avg}}\, \theta_{t+1} - \lambda\, \bar\theta_t.

Before t0_ the average simply tracks $\theta$ ( $\bar\theta = \theta$ ).

Math

Per-parameter update (post warm-up):

g_t \leftarrow g_t + \lambda\, \theta_t,

v_{t+1} = \mu\, v_t + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1},

\bar\theta_{t+1} = (1 - \eta_{\mathrm{avg}})\, \bar\theta_t + \eta_{\mathrm{avg}}\, \theta_{t+1} - \lambda\, \bar\theta_t.

References

Polyak and Juditsky, "Acceleration of stochastic approximation by averaging" (SIAM J. Control Optim., 1992).

See Also

SGD : the underlying instantaneous update.

Attributes

lr_double

Learning rate

\eta

momentum_double

Momentum coefficient

\mu

. Zero disables velocity allocation.

weight_decay_double

L2 penalty coefficient

\lambda

alpha_double

Decay exponent controlling the averaging schedule (

\eta_{\mathrm{avg}} = 1 / (\alpha t + 1)

t0_double

Number of warm-up steps before averaging starts. Stored as double to mirror the reference framework's API but compared against integer step counts.

lambd_double

Average-side decay coefficient applied to

\bar\theta

each averaging step.

moment_std::vector<Storage>

Per-parameter SGD velocity buffers (active when momentum_ != 0).

ax_std::vector<Storage>

Per-parameter running averages of the parameter trajectory. Initialised to a copy of the parameter on the first observed gradient.

step_std::vector<std::int64_t>

Per-parameter step counters driving

\eta_{\mathrm{avg}}

Notes

The step counter is maintained per parameter slot rather than as a single global counter so that parameters introduced into training late (or temporarily frozen) still see a clean averaging schedule.

Constructors

ctor

ASGD

void ASGD(int)

source edit

Construct an ASGD optimizer.

Parameters

paramsstd::vector<std::shared_ptr<TensorImpl>>

Parameters to optimise.

lrdouble= None

Learning rate

\eta

(default 1e-3).

momentumdouble= None

SGD momentum coefficient

\mu

(default 0.0).

weight_decaydouble= None

L2 regularisation coefficient

\lambda

(default 0.0).

alphadouble= None

Decay exponent for the averaging schedule (default 0.75).

t0double= None

Warm-up step count before averaging engages (default 1e6).

lambddouble= None

Average-side decay coefficient (default 1e-4).

Virtual methods

virtconst

lr

double lr()

source edit

Current learning rate $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Update the learning rate from a scheduler.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate per-slot state on the first observed gradient.

Materialises the velocity buffer (only when momentum_ != 0) and the running-average buffer ax_ initialised to a copy of the current parameter. The per-slot step counter is reset to 0.

Parameters

istd::size_t

Slot index.

pconst std::shared_ptr<TensorImpl>&

Parameter whose layout dictates the state buffer shapes.

fnconst

state_dict_id

int state_dict_id()

source edit

Checkpoint identifier ("asgd_v1").

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply the ASGD update for one parameter slot.

Parameters

istd::size_t

Slot index into params_, moment_, ax_, step_.

pstd::shared_ptr<TensorImpl>&

Parameter to update in place.

gconst Storage&

Accumulated gradient for this step.