selu

→Tensor

selu(x: Tensor, inplace: bool = False)

source

Scaled exponential linear unit activation.

Self-normalising activation from Klambauer et al. (2017): when used in a fully-connected network with LeCun-normal initialised weights, the fixed scale $\lambda$ and slope $\alpha$ drive the activations toward zero-mean, unit-variance fixed points without the need for explicit batch / layer normalisation.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

inplacebool= False

Accepted for API compatibility; currently ignored.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{SELU}(x) = \lambda \, \text{ELU}(x, \alpha) = \lambda \begin{cases} x & x > 0 \\ \alpha (e^x - 1) & x \le 0 \end{cases}

with the (non-tunable) constants

\lambda \approx 1.0507, \qquad \alpha \approx 1.6733,

chosen so that the activation has a zero-mean unit-variance attractor fixed point. Pair with alpha_dropout and LeCun-normal weight init to retain the self-normalising property.

Examples

>>> import lucid
>>> from lucid.nn.functional import selu
>>> x = lucid.tensor([-1.0, 0.0, 1.0])
>>> selu(x)
Tensor([-1.1113,  0.0000,  1.0507])