class

Rprop

extendsOptimizer

Rprop

source edit

Rprop optimiser — Resilient Backpropagation (Riedmiller & Braun 1993).

A magnitude-free quasi-Newton method: each parameter carries its own step size which grows when the gradient sign is consistent and shrinks when it flips. Because only $\mathrm{sign}(g)$ enters the update, Rprop is immune to gradient-magnitude pathologies (vanishing or exploding) but is incompatible with mini-batch noise — it is a batch-mode method.

Math

Let $\Delta_t$ be the per-parameter step size and $g_t$ the gradient.

\Delta_{t+1} = \begin{cases} \min(\eta_+\,\Delta_t,\;\Delta_{\max}) & g_t\,g_{t-1} > 0 \\ \max(\eta_-\,\Delta_t,\;\Delta_{\min}) & g_t\,g_{t-1} < 0 \\ \Delta_t & g_t\,g_{t-1} = 0 \end{cases}

When the sign flips, $g_t$ is additionally zeroed to suppress a secondary overshoot on the following step. The parameter update is

\theta_{t+1} = \theta_t - \mathrm{sign}(g_t)\,\Delta_{t+1}

References

Riedmiller & Braun, "A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm" (ICNN 1993).

See Also

RMSprop — Hinton's mini-batch-friendly replacement that still uses an EMA of $g^2$ but keeps the gradient magnitude in the update.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 1e-2

Initial value placed into every element of step_size_. Despite the name this is not used as a global scale.

eta_minusfloat, default 0.5

Step-shrink factor

\eta_-

applied on sign flips.

eta_plusfloat, default 1.2

Step-grow factor

\eta_+

applied on consistent signs.

step_minfloat, default 1e-6

Lower clamp

\Delta_{\min}

step_maxfloat, default 50.0

Upper clamp

\Delta_{\max}

Attributes

prev_grad_vector of Storage

Last-step gradient, used to detect sign flips.

step_size_vector of Storage

Per-element adaptive step

\Delta

. Initialised to lr_ on every element rather than to 1.0, so the very first update has a sensible scale.

Notes

Rprop assumes deterministic gradients (full-batch). Applying it to mini-batches confuses sign-flip detection because batch noise causes spurious sign changes — accepted practice is to either use very large batches or switch to RMSprop.

Constructors

ctor

Rprop

void Rprop(int)

source edit

Construct the optimiser and resize per-slot state vectors.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 1e-2

Initial value for every element of step_size_.

eta_minusfloat, default 0.5

Step-shrink factor.

eta_plusfloat, default 1.2

Step-grow factor.

step_minfloat, default 1e-6

Lower step clamp.

step_maxfloat, default 50.0

Upper step clamp.

Virtual methods

virtconst

lr

→float

double lr()

source edit

Return the current lr_ parameter.

Returns

float

Latest value passed to set_lr (or the constructor).

virt

set_lr

void set_lr(double lr)

source edit

Set the active learning-rate parameter.

Parameters

lrfloat

New lr_ value. This is not retroactively applied to existing step_size_ buffers — it only affects fresh allocations.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate prev_grad_ and step_size_ for parameter i.

Parameters

iint

Slot index.

pTensorImpl

Parameter whose shape/dtype/device dictates the buffer layout.

Notes

prev_grad_[i] is zero-initialised; step_size_[i] is filled with lr_ so the first step has a sensible magnitude.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Return the versioned state-dict identifier.

Returns

str

"rprop_v1".

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply one Rprop update to parameter i.

Parameters

iint

Parameter slot index.

pTensorImpl

Parameter tensor (updated in place).

gStorage

Gradient for p on this step. May be zeroed in place where the sign of g flipped relative to prev_grad_.

Notes

Drives the per-element step-size adaptation rule above and then applies $-\mathrm{sign}(g)\,\Delta$ to p.

class

Rprop

extendsOptimizer

Rprop

source edit

Rprop optimiser — Resilient Backpropagation (Riedmiller & Braun 1993).

Math

Let $\Delta_t$ be the per-parameter step size and $g_t$ the gradient.

\Delta_{t+1} = \begin{cases} \min(\eta_+\,\Delta_t,\;\Delta_{\max}) & g_t\,g_{t-1} > 0 \\ \max(\eta_-\,\Delta_t,\;\Delta_{\min}) & g_t\,g_{t-1} < 0 \\ \Delta_t & g_t\,g_{t-1} = 0 \end{cases}

When the sign flips, $g_t$ is additionally zeroed to suppress a secondary overshoot on the following step. The parameter update is

\theta_{t+1} = \theta_t - \mathrm{sign}(g_t)\,\Delta_{t+1}

References

Riedmiller & Braun, "A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm" (ICNN 1993).

See Also

RMSprop — Hinton's mini-batch-friendly replacement that still uses an EMA of $g^2$ but keeps the gradient magnitude in the update.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 1e-2

Initial value placed into every element of step_size_. Despite the name this is not used as a global scale.

eta_minusfloat, default 0.5

Step-shrink factor

\eta_-

applied on sign flips.

eta_plusfloat, default 1.2

Step-grow factor

\eta_+

applied on consistent signs.

step_minfloat, default 1e-6

Lower clamp

\Delta_{\min}

step_maxfloat, default 50.0

Upper clamp

\Delta_{\max}

Attributes

prev_grad_vector of Storage

Last-step gradient, used to detect sign flips.

step_size_vector of Storage

Per-element adaptive step

\Delta

. Initialised to lr_ on every element rather than to 1.0, so the very first update has a sensible scale.

Notes

Constructors

ctor

Rprop

void Rprop(int)

source edit

Construct the optimiser and resize per-slot state vectors.

Parameters

paramsvector of TensorImpl

Parameters to optimise.

lrfloat, default 1e-2

Initial value for every element of step_size_.

eta_minusfloat, default 0.5

Step-shrink factor.

eta_plusfloat, default 1.2

Step-grow factor.

step_minfloat, default 1e-6

Lower step clamp.

step_maxfloat, default 50.0

Upper step clamp.

Virtual methods

virtconst

lr

→float

double lr()

source edit

Return the current lr_ parameter.

Returns

float

Latest value passed to set_lr (or the constructor).

virt

set_lr

void set_lr(double lr)

source edit

Set the active learning-rate parameter.

Parameters

lrfloat

New lr_ value. This is not retroactively applied to existing step_size_ buffers — it only affects fresh allocations.

Methods

init_state_slot

void init_state_slot(int i, const int & p)

source edit

Allocate prev_grad_ and step_size_ for parameter i.

Parameters

iint

Slot index.

pTensorImpl

Parameter whose shape/dtype/device dictates the buffer layout.

Notes

prev_grad_[i] is zero-initialised; step_size_[i] is filled with lr_ so the first step has a sensible magnitude.

fnconst

state_dict_id

→str

int state_dict_id()

source edit

Return the versioned state-dict identifier.

Returns

str

"rprop_v1".

update_one

void update_one(int i, int & p, const int & g)

source edit

Apply one Rprop update to parameter i.

Parameters

iint

Parameter slot index.

pTensorImpl

Parameter tensor (updated in place).

gStorage

Gradient for p on this step. May be zeroed in place where the sign of g flipped relative to prev_grad_.

Notes

Drives the per-element step-size adaptation rule above and then applies $-\mathrm{sign}(g)\,\Delta$ to p.