Stochastic gradient descent with optional momentum, Nesterov acceleration, and L2 weight decay.
Plain SGD takes the parameter down the negative gradient direction at fixed learning rate :
With momentum coefficient , a per-parameter velocity buffer accumulates a low-pass-filtered gradient (Polyak's heavy ball):
where the dampening coefficient scales the current gradient
when accumulating into . dampening = 0 recovers classical
Polyak momentum.
With Nesterov acceleration (Sutskever et al. 2013 reformulation), the lookahead form is used:
which requires dampening = 0 and momentum > 0.
L2 weight decay with coefficient is applied to the gradient before the momentum update (PyTorch-style coupled weight decay):
This is the standard L2 regularisation form; AdamW-style decoupled
weight decay is not used here — use AdamW for that variant.
Math
Full update (momentum branch, no Nesterov):
Plain SGD is the special case where the velocity buffer is never allocated.
References
Polyak, "Some methods of speeding up the convergence of iteration methods" (1964). Sutskever, Martens, Dahl, Hinton, "On the importance of initialization and momentum in deep learning" (ICML 2013).
See Also
ASGD : averaged SGD with running parameter mean. Adam, AdamW : adaptive moment optimisers.
Attributes
lr_doubleset_lr.momentum_doubledampening_doublenesterov_ is true.weight_decay_doublenesterov_boolmomentum_ > 0 and dampening_ == 0.moment_std::vector<Storage>init_state_slot on the first step seen by slot only when momentum_ != 0; otherwise it remains empty.Notes
CPU and GPU update paths are dispatched inside update_one based
on the parameter's device. The CPU path is a flat scalar loop over
the raw byte buffer; the GPU path expresses the update as a few MLX
array ops, which compose with the surrounding lazy graph.
Examples
Typical training-loop usage from Python:>>> opt = lucid.optim.SGD(model.parameters(), lr=0.01,
... momentum=0.9, weight_decay=1e-4)
>>> opt.zero_grad()
>>> loss.backward()
>>> opt.step()Constructors
1Construct an SGD optimizer with optional momentum and Nesterov.
Parameters
paramsstd::vector<std::shared_ptr<TensorImpl>>Optimizer.lrdoublemomentumdouble= None0.0). Zero disables momentum.dampeningdouble= None0.0). Must be 0 when nesterov is true.weight_decaydouble= None0.0).nesterovbool= Nonemomentum > 0 and dampening == 0.Raises
:runtime_errornesterov is requested without positive momentum, or with a non-zero dampening.Virtual methods
2Methods
7Allocate the velocity buffer for one slot when momentum != 0.
Parameters
slot_idxstd::size_tparams_ and moment_.paramconst std::shared_ptr<TensorImpl>&Notes
When momentum_ == 0 the corresponding moment_ entry is
left empty so that plain SGD allocates no extra memory.
Restore the velocity buffers from a checkpoint snapshot.
Parameters
bufsconst std::vector<NamedBuffers>&"momentum_buffer" and whose tensor list matches the layout of state_buffers.Raises
:runtime_errorCurrent momentum coefficient .
Snapshot the per-parameter velocity buffers for checkpointing.
See Also
load_state_buffers : the inverse operation.
Returns
std::vector<NamedBuffers>Single-entry list [("momentum_buffer", tensors)] whose tensors runs parallel to params_. Slots without an allocated velocity (momentum == 0 or the slot has never received a gradient) contribute a null pointer.
Checkpoint identifier ("sgd_v1").
Apply the SGD update for one parameter slot.
Dispatches to either the CPU scalar loop or the MLX GPU path based on the parameter's device. Implements the plain-SGD, momentum, Nesterov, and weight-decay branches according to the constructor flags. See the class-level math block for the precise update rule.
Parameters
slot_idxstd::size_tparams_ and moment_.paramstd::shared_ptr<TensorImpl>&gradconst Storage&Current L2 weight-decay coefficient .