rrelu

→Tensor

rrelu(x: Tensor, lower: float = 1.0 / 8.0, upper: float = 1.0 / 3.0, training: bool = False, inplace: bool = False)

source

Randomised leaky ReLU (Xu et al. 2015).

A regularised variant of leaky_relu: during training every negative element is scaled by an i.i.d. uniform slope, injecting activation noise on the negative side. At evaluation time the expectation of that uniform draw is used, so the function becomes deterministic.

Parameters

xTensor

Input tensor.

lowerfloat= 1.0 / 8.0

Lower bound of the uniform slope distribution. Default 1/8.

upperfloat= 1.0 / 3.0

Upper bound of the uniform slope distribution. Default 1/3.

trainingbool= False

If True sample a fresh slope per element; if False use the midpoint. Default False.

inplacebool= False

Accepted for API compatibility; currently ignored.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

During training, with $a \sim \mathrm{Uniform}(\text{lower}, \text{upper})$ sampled element-wise:

\text{RReLU}(x) = \begin{cases} x & x \ge 0 \\ a x & x < 0 \end{cases}

During evaluation, $a$ is replaced by the expectation $(\text{lower} + \text{upper}) / 2$ . The slope randomisation acts as a mild noise regulariser — comparable in spirit to dropout but applied to activation slope rather than activation magnitude.

Examples

>>> import lucid
>>> from lucid.nn.functional import rrelu
>>> x = lucid.tensor([-1.0, 0.0, 1.0])
>>> rrelu(x, training=False)
Tensor([-0.2292,  0.0000,  1.0000])