class

FeatureAlphaDropout

extendsModule

FeatureAlphaDropout(p: float = 0.5, inplace: bool = False)

source

Channel-wise alpha dropout that preserves SELU self-normalisation.

Combines the ideas of Dropout2d (zero entire channels) and AlphaDropout (maintain zero mean / unit variance after masking). A single Bernoulli draw per channel per sample decides whether that entire feature map is replaced by the SELU saturation value $\alpha'$ ; an affine correction is then applied to restore the distributional invariants:

y_{n,c} = \begin{cases} x_{n,c} & \text{with probability } 1 - p \\ \alpha' \cdot \mathbf{1} & \text{with probability } p \end{cases}

\tilde{y}_{n,c} = a \cdot y_{n,c} + b

where the spatial dimensions are suppressed for clarity, $\alpha' = -\lambda\alpha \approx -1.7581$ , and the affine coefficients $a, b$ restore $\mathbb{E}[\tilde{y}] = 0$ , $\operatorname{Var}[\tilde{y}] = 1$ across the channel.

This variant is appropriate for SELU-activated convolutional networks, where using AlphaDropout on individual pixels would be too local (adjacent pixels are correlated) while Dropout2d would break the self-normalising statistics.

Parameters

pfloat= 0.5

Probability of replacing an entire channel with

\alpha'

. Must be in [0, 1]. Default: 0.5.

inplacebool= False

Accepted for API compatibility; has no effect. Default: False.

Notes

Input: (N, C, *) — any number of spatial dimensions.
Output: same shape (N, C, *).

The mask is sampled on the (N, C) axes and broadcast over all remaining dimensions, so the full spatial volume of each channel is either kept intact or replaced uniformly.

In eval mode the layer is the identity.

Use this module only after lucid.nn.SELU activations. Applying it after non-SELU activations yields no statistical benefit.

AlphaDropout : Element-wise variant for SELU networks. Dropout2d : Channel-wise dropout without the SELU correction.

Examples

In a self-normalising convolutional network:
>>> import lucid, lucid.nn as nn
>>> block = nn.Sequential(
...     nn.Conv2d(16, 32, kernel_size=3, padding=1),
...     nn.SELU(),
...     nn.FeatureAlphaDropout(p=0.05),
... )
>>> block.train()
>>> y = block(lucid.randn(2, 16, 8, 8))
>>> y.shape
(2, 32, 8, 8)
Compare channel mask vs. element mask for SELU conv features:
>>> fad = nn.FeatureAlphaDropout(p=0.3)
>>> fad.train()
>>> x = lucid.randn(4, 8, 6, 6)
>>> out = fad(x)
>>> out.shape
(4, 8, 6, 6)

Methods (3)

dunder

init

→None

__init__(p: float = 0.5, inplace: bool = False)

source

Initialise the FeatureAlphaDropout module. See the class docstring for parameter semantics.

forward

→Tensor

forward(x: Tensor)

source

Apply dropout to the input tensor.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input; in eval mode this is the identity.

extra_repr

→str

extra_repr()

source

Return a string representation of the layer's configuration.