class

NAdam

extendsOptimizer

NAdam

source edit

NAdam optimiser — Adam with Nesterov momentum (Dozat, 2016).

Replaces the bias-corrected first-moment term with a Nesterov look-ahead that blends the current gradient $g_t$ and the running moment $m_t$ . The momentum coefficient $\mu_t$ decays each step, and its running product $\prod_{s \le t} \mu_s$ replaces the $1 - \beta_1^t$ bias correction.

Math

\mu_t = \beta_1 \left( 1 - \tfrac{1}{2} \cdot 0.96^{t \psi} \right), \qquad \mu_{t+1} = \beta_1 \left( 1 - \tfrac{1}{2} \cdot 0.96^{(t+1)\psi} \right)

where $\psi$ is momentum_decay. Let $\Pi_t = \prod_{s=1}^{t} \mu_s$ and $\Pi_{t+1} = \Pi_t \cdot \mu_{t+1}$ .

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

d_t = \sqrt{v_t / (1 - \beta_2^t)} + \epsilon

\theta_{t+1} = \theta_t - \eta\, \frac{(1 - \mu_t)}{1 - \Pi_t} \cdot \frac{g_t}{d_t} - \eta\, \frac{\mu_{t+1}}{1 - \Pi_{t+1}} \cdot \frac{m_t}{d_t}

References

Dozat, "Incorporating Nesterov Momentum into Adam" (ICLR Workshop, 2016).

See Also

Adam, AdamW, RAdam.

Attributes

lr_float

Step size

\eta

. Default 2e-3 (paper recommendation).

beta1_, beta2_, eps_, weight_decay_float

As in Adam.

momentum_decay_float

Decay constant

\psi

in the

\mu_t

schedule. Default 4e-3.

m_, v_Storage[]

Per-parameter first- and second-moment buffers.

mu_product_double[]

Running product

\Pi_t

of all momentum coefficients seen so far, one scalar per parameter.

step_count_int64

Global step counter.

Notes

mu_product_ is a per-parameter double, not a Storage, because it is a single scalar accumulator rather than a tensor. Weight decay (if non-zero) is added to $g_t$ in the standard L2 form — there is no published "decoupled NAdam".

Constructors

ctor

NAdam

void NAdam(int)

source edit

Construct a NAdam optimiser bound to params.

Parameters

paramsvector of TensorImpl shared pointers

Parameters to optimise.

lrfloat= None

Step size

\eta

. Default 2e-3.

beta1, beta2float= None

EMA decay rates. Defaults 0.9 / 0.999.

epsfloat= None

Denominator stabiliser. Default 1e-8.

weight_decayfloat= None

L2 penalty coefficient. Default 0.

momentum_decayfloat= None

Momentum schedule decay

\psi

. Default 4e-3.

Virtual methods

virtconst

lr

double lr()

source edit

Current learning rate $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Set the learning rate.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate zero-initialised $m$ and $v$ buffers; initialise mu_product_[i] to 1.0.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Schema identifier for state_dict serialisation.

Returns

str

"nadam_v1".

update_one

void update_one(int i, int & p, const int & g)

source edit

Compute per-step $\mu_t$ and $\mu_{t+1}$ , advance mu_product_, then apply the two-term Nesterov update.

Parameters

isize_t

Slot index into m_ / v_ / mu_product_.

pTensorImpl shared pointer

In-place parameter.

gStorage

Gradient

g_t

class

NAdam

extendsOptimizer

NAdam

source edit

NAdam optimiser — Adam with Nesterov momentum (Dozat, 2016).

Math

\mu_t = \beta_1 \left( 1 - \tfrac{1}{2} \cdot 0.96^{t \psi} \right), \qquad \mu_{t+1} = \beta_1 \left( 1 - \tfrac{1}{2} \cdot 0.96^{(t+1)\psi} \right)

where $\psi$ is momentum_decay. Let $\Pi_t = \prod_{s=1}^{t} \mu_s$ and $\Pi_{t+1} = \Pi_t \cdot \mu_{t+1}$ .

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

d_t = \sqrt{v_t / (1 - \beta_2^t)} + \epsilon

\theta_{t+1} = \theta_t - \eta\, \frac{(1 - \mu_t)}{1 - \Pi_t} \cdot \frac{g_t}{d_t} - \eta\, \frac{\mu_{t+1}}{1 - \Pi_{t+1}} \cdot \frac{m_t}{d_t}

References

Dozat, "Incorporating Nesterov Momentum into Adam" (ICLR Workshop, 2016).

See Also

Adam, AdamW, RAdam.

Attributes

lr_float

Step size

\eta

. Default 2e-3 (paper recommendation).

beta1_, beta2_, eps_, weight_decay_float

As in Adam.

momentum_decay_float

Decay constant

\psi

in the

\mu_t

schedule. Default 4e-3.

m_, v_Storage[]

Per-parameter first- and second-moment buffers.

mu_product_double[]

Running product

\Pi_t

of all momentum coefficients seen so far, one scalar per parameter.

step_count_int64

Global step counter.

Notes

Constructors

ctor

NAdam

void NAdam(int)

source edit

Construct a NAdam optimiser bound to params.

Parameters

paramsvector of TensorImpl shared pointers

Parameters to optimise.

lrfloat= None

Step size

\eta

. Default 2e-3.

beta1, beta2float= None

EMA decay rates. Defaults 0.9 / 0.999.

epsfloat= None

Denominator stabiliser. Default 1e-8.

weight_decayfloat= None

L2 penalty coefficient. Default 0.

momentum_decayfloat= None

Momentum schedule decay

\psi

. Default 4e-3.

Virtual methods

virtconst

lr

double lr()

source edit

Current learning rate $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Set the learning rate.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate zero-initialised $m$ and $v$ buffers; initialise mu_product_[i] to 1.0.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Schema identifier for state_dict serialisation.

Returns

str

"nadam_v1".

update_one

void update_one(int i, int & p, const int & g)

source edit

Compute per-step $\mu_t$ and $\mu_{t+1}$ , advance mu_product_, then apply the two-term Nesterov update.

Parameters

isize_t

Slot index into m_ / v_ / mu_product_.

pTensorImpl shared pointer

In-place parameter.

gStorage

Gradient

g_t