calculate_gain

→float

calculate_gain(nonlinearity: str, param: float | None = None)

source

Return the recommended variance-preserving gain for an activation.

The gain is a multiplicative correction applied to the standard deviation of Xavier / Kaiming initialisation so the per-layer activation variance is preserved after the nonlinearity. Values follow the original recommendations in Glorot & Bengio (2010) and He et al. (2015).

Parameters

nonlinearitystr

Name of the activation function. One of 'linear', 'conv1d', 'conv2d', 'conv3d', 'sigmoid', 'tanh', 'relu', 'leaky_relu', 'selu'.

paramfloat= None

Negative slope of the rectifier — only consulted when nonlinearity='leaky_relu'. Default 0.01.

Returns

float

Recommended gain factor.

Raises

ValueError

If nonlinearity is not in the supported set.

Notes

Mapping table:

=============== ============================================ nonlinearity gain =============== ============================================ linear $1$ conv{1,2,3}d $1$ sigmoid $1$ tanh $5/3 \approx 1.667$ relu $\sqrt{2} \approx 1.414$ leaky_relu $\sqrt{2 / (1 + a^2)}$ selu $3/4$ =============== ============================================

Examples

>>> from lucid.nn.init import calculate_gain
>>> calculate_gain('relu')
1.4142135623730951
>>> calculate_gain('leaky_relu', 0.2)
1.3867504905630728