class

Adamax

extendsOptimizer

Adamax

source edit

Adamax optimiser — Adam variant using the infinity norm.

Replaces Adam's $\ell_2$ second-moment estimate with an exponentially decayed running maximum, giving a bounded effective step size that is well-behaved when occasional gradient components are very large.

Math

m_{t+1} = \beta_1 m_t + (1 - \beta_1) g_t

u_{t+1} = \max(\beta_2 u_t,\; |g_t|)

\theta_{t+1} = \theta_t - \frac{\eta}{1 - \beta_1^{t+1}} \cdot \frac{m_{t+1}}{u_{t+1} + \epsilon}

The second-moment recursion is the limit of an Adam-style $L^p$ EMA as $p \to \infty$ , hence "max".

References

Kingma & Ba, "Adam: A Method for Stochastic Optimization", §7.1 "Adamax" (ICLR 2015).

See Also

Adagrad, Adadelta, RMSprop

Parameters

paramsvector of TensorImpl

Parameters to optimise; one update buffer per entry.

lrfloat, default 2e-3

Base step size

\eta

beta1float, default 0.9

First-moment EMA decay.

beta2float, default 0.999

Decay used inside the running-max envelope.

epsfloat, default 1e-8

Numerical stabiliser added to

u

weight_decayfloat, default 0.0

L2 penalty applied to

g_t

before the update.

Attributes

m_vector of Storage

Per-parameter first-moment EMA.

u_vector of Storage

Per-parameter

\ell_\infty

envelope (the "infinity-norm" buffer).

step_count_int

Global step counter, used for bias correction of

m

Notes

Unlike Adam there is no bias correction on $u$ — the running max is naturally bounded, so dividing it by $1 - \beta_2^{t+1}$ would over-amplify the very first updates.

Constructors

ctor

Adamax

void Adamax(int)

source edit

Construct the optimiser and resize per-slot state vectors.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 2e-3

Base step size.

beta1float, default 0.9

First-moment EMA decay.

beta2float, default 0.999

Infinity-norm envelope decay.

epsfloat, default 1e-8

Numerical stabiliser.

weight_decayfloat, default 0.0

L2 penalty coefficient.

Virtual methods

virtconst

lr

→float

double lr()

source edit

Return the current learning rate.

Returns

float

Active value of $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Set the active learning rate.

Parameters

lrfloat

New base step size

\eta

used on subsequent steps.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate the per-slot state buffers for parameter i.

Parameters

iint

Slot index.

pTensorImpl

Parameter whose shape/dtype/device dictates the buffer layout.

Notes

Both m_[i] (first moment) and u_[i] (infinity norm) are zero-initialised.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Return the versioned state-dict identifier.

Returns

str

"adamax_v1" — used by serialisation to validate state-dict layout.

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply one Adamax update to parameter i.

Parameters

iint

Index of the parameter in the optimiser's parameter list.

pTensorImpl

Parameter tensor (updated in place).

gStorage

Gradient for p on this step.

Notes

Advances $m$ via the EMA, refreshes the $\ell_\infty$ envelope $u$ , then applies the bias-corrected step in one fused pass.

class

Adamax

extendsOptimizer

Adamax

source edit

Adamax optimiser — Adam variant using the infinity norm.

Math

m_{t+1} = \beta_1 m_t + (1 - \beta_1) g_t

u_{t+1} = \max(\beta_2 u_t,\; |g_t|)

\theta_{t+1} = \theta_t - \frac{\eta}{1 - \beta_1^{t+1}} \cdot \frac{m_{t+1}}{u_{t+1} + \epsilon}

The second-moment recursion is the limit of an Adam-style $L^p$ EMA as $p \to \infty$ , hence "max".

References

Kingma & Ba, "Adam: A Method for Stochastic Optimization", §7.1 "Adamax" (ICLR 2015).

See Also

Adagrad, Adadelta, RMSprop

Parameters

paramsvector of TensorImpl

Parameters to optimise; one update buffer per entry.

lrfloat, default 2e-3

Base step size

\eta

beta1float, default 0.9

First-moment EMA decay.

beta2float, default 0.999

Decay used inside the running-max envelope.

epsfloat, default 1e-8

Numerical stabiliser added to

u

weight_decayfloat, default 0.0

L2 penalty applied to

g_t

before the update.

Attributes

m_vector of Storage

Per-parameter first-moment EMA.

u_vector of Storage

Per-parameter

\ell_\infty

envelope (the "infinity-norm" buffer).

step_count_int

Global step counter, used for bias correction of

m

Notes

Unlike Adam there is no bias correction on $u$ — the running max is naturally bounded, so dividing it by $1 - \beta_2^{t+1}$ would over-amplify the very first updates.

Constructors

ctor

Adamax

void Adamax(int)

source edit

Construct the optimiser and resize per-slot state vectors.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 2e-3

Base step size.

beta1float, default 0.9

First-moment EMA decay.

beta2float, default 0.999

Infinity-norm envelope decay.

epsfloat, default 1e-8

Numerical stabiliser.

weight_decayfloat, default 0.0

L2 penalty coefficient.

Virtual methods

virtconst

lr

→float

double lr()

source edit

Return the current learning rate.

Returns

float

Active value of $\eta$ .

virt

set_lr

void set_lr(double lr)

source edit

Set the active learning rate.

Parameters

lrfloat

New base step size

\eta

used on subsequent steps.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate the per-slot state buffers for parameter i.

Parameters

iint

Slot index.

pTensorImpl

Parameter whose shape/dtype/device dictates the buffer layout.

Notes

Both m_[i] (first moment) and u_[i] (infinity norm) are zero-initialised.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Return the versioned state-dict identifier.

Returns

str

"adamax_v1" — used by serialisation to validate state-dict layout.

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply one Adamax update to parameter i.

Parameters

iint

Index of the parameter in the optimiser's parameter list.

pTensorImpl

Parameter tensor (updated in place).

gStorage

Gradient for p on this step.

Notes

Advances $m$ via the EMA, refreshes the $\ell_\infty$ envelope $u$ , then applies the bias-corrected step in one fused pass.