alpha_dropout

→Tensor

alpha_dropout(x: Tensor, p: float = 0.5, training: bool = True, inplace: bool = False)

source

Alpha dropout — variance-preserving dropout for SELU networks.

Designed to be paired with the SELU activation (Klambauer et al. 2017 "Self-Normalizing Neural Networks"). Standard dropout breaks the carefully-tuned zero-mean / unit-variance propagation that makes SELU networks self-normalising; alpha dropout fixes this by replacing dropped activations with the SELU saturation value $\alpha' = -\alpha \cdot \text{scale}$ (rather than zero), then applying an affine rescaling chosen so that the post-dropout activations again have zero mean and unit variance.

Parameters

xTensor

Input tensor, any shape.

pfloat= 0.5

Element-drop probability in

[0, 1]

(default 0.5).

trainingbool= True

When False, identity (default True).

inplacebool= False

Reserved for API parity; the implementation always produces a new tensor (default False).

Returns

Tensor

Same shape and dtype as x.

Notes

With keep probability $q = 1 - p$ , each element is independently set to its original value (with probability $q$ ) or to $\alpha'$ (with probability $p$ ). The result is then affine-transformed:

y = a \cdot \mathrm{mix}(x, \alpha') + b, \quad a = \big(q + p\,(\alpha')^2\,q\big)^{-1/2}, \quad b = -a\,\alpha'\,p,

chosen so that $\mathbb{E}[y] = 0$ and $\mathrm{Var}(y) = 1$ when the input has zero mean and unit variance.

Examples

>>> import lucid
>>> from lucid.nn.functional import alpha_dropout
>>> x = lucid.randn(8)
>>> y = alpha_dropout(x, p=0.1, training=True)