kaiming_normal_

→Tensor

kaiming_normal_(tensor: Tensor, a: float = 0, mode: str = 'fan_in', nonlinearity: str = 'leaky_relu')

source

Initialise tensor in-place with Kaiming (He) normal initialisation.

Gaussian counterpart of kaiming_uniform_. Draws each entry from $\mathcal{N}(0, \sigma^2)$ with a standard deviation chosen to preserve activation (or gradient) variance across a stack of ReLU-family layers, as proposed in He et al. (2015).

Parameters

tensorTensor

Tensor to initialise in place; must have at least 2 dimensions.

afloat= 0

Negative slope of the rectifier (only used when nonlinearity='leaky_relu'). Default 0.

mode(fan_in, fan_out)= 'fan_in'

'fan_in' preserves forward-pass variance, 'fan_out' preserves backward-pass gradient variance. Default 'fan_in'.

nonlinearitystr= 'leaky_relu'

Activation name forwarded to calculate_gain. Default 'leaky_relu'.

Returns

Tensor

tensor (mutated) for chaining.

Notes

With $n = \text{fan}$ the standard deviation is

\sigma = \frac{\text{gain}}{\sqrt{n}},

so that for ReLU $\mathrm{Var}(W) = 2 / n$ .

Examples

>>> import lucid
>>> from lucid.nn.init import kaiming_normal_
>>> w = lucid.empty(64, 32)
>>> kaiming_normal_(w, nonlinearity='relu')