FeatureAlphaDropout
ModuleFeatureAlphaDropout(p: float = 0.5, inplace: bool = False)Channel-wise alpha dropout that preserves SELU self-normalisation.
Combines the ideas of Dropout2d (zero entire channels) and
AlphaDropout (maintain zero mean / unit variance after
masking). A single Bernoulli draw per channel per sample decides
whether that entire feature map is replaced by the SELU saturation
value ; an affine correction is then applied to
restore the distributional invariants:
where the spatial dimensions are suppressed for clarity, , and the affine coefficients restore , across the channel.
This variant is appropriate for SELU-activated convolutional
networks, where using AlphaDropout on individual pixels
would be too local (adjacent pixels are correlated) while
Dropout2d would break the self-normalising statistics.
Parameters
pfloat= 0.5[0, 1]. Default: 0.5.inplacebool= FalseFalse.Notes
- Input:
(N, C, *)— any number of spatial dimensions. - Output: same shape
(N, C, *).
The mask is sampled on the (N, C) axes and broadcast over all
remaining dimensions, so the full spatial volume of each channel is
either kept intact or replaced uniformly.
In eval mode the layer is the identity.
Use this module only after lucid.nn.SELU activations.
Applying it after non-SELU activations yields no statistical benefit.
Examples
In a self-normalising convolutional network:
>>> import lucid, lucid.nn as nn
>>> block = nn.Sequential(
... nn.Conv2d(16, 32, kernel_size=3, padding=1),
... nn.SELU(),
... nn.FeatureAlphaDropout(p=0.05),
... )
>>> block.train()
>>> y = block(lucid.randn(2, 16, 8, 8))
>>> y.shape
(2, 32, 8, 8)
Compare channel mask vs. element mask for SELU conv features:
>>> fad = nn.FeatureAlphaDropout(p=0.3)
>>> fad.train()
>>> x = lucid.randn(4, 8, 6, 6)
>>> out = fad(x)
>>> out.shape
(4, 8, 6, 6)See Also
AlphaDropout—Element-wise variant for SELU networks.Dropout2d—Channel-wise dropout without the SELU correction.
Used by 1
Constructors
1Instance methods
2Return a string representation of the layer's configuration.