class

RReLU

extendsModule

RReLU(lower: float = 1.0 / 8.0, upper: float = 1.0 / 3.0, inplace: bool = False)

source

Randomized Leaky ReLU activation function.

During training, the negative slope for each element is drawn independently and uniformly from $[\text{lower}, \text{upper}]$ :

\text{RReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ a \cdot x & \text{otherwise},\quad a \sim \mathcal{U}(\text{lower},\, \text{upper}) \end{cases}

During evaluation, the slope is fixed at the deterministic midpoint $a = (\text{lower} + \text{upper}) / 2$ .

The stochastic negative slope acts as a form of noise-based regularisation, similar in spirit to Dropout, and was found to improve generalisation in image classification tasks.

Parameters

lowerfloat= 1.0 / 8.0

Lower bound of the uniform slope distribution. Default: 1/8.

upperfloat= 1.0 / 3.0

Upper bound of the uniform slope distribution. Default: 1/3.

inplacebool= False

If True, modifies the input tensor in-place. Default: False.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

Call m.train() / m.eval() to switch between stochastic and deterministic modes respectively. The module inherits training-mode tracking from lucid.nn.Module.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.RReLU(lower=0.1, upper=0.3)
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> out_train = m(x)          # slope sampled from U(0.1, 0.3)
>>> m.eval()
>>> out_eval = m(x)           # slope fixed at 0.2
>>> out_eval.shape
(5,)

Methods (3)

dunder

init

→None

__init__(lower: float = 1.0 / 8.0, upper: float = 1.0 / 3.0, inplace: bool = False)

source

Initialise the RReLU module. See the class docstring for parameter semantics.

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.

extra_repr

→str

extra_repr()

source

Return a string representation of the layer's configuration.