xavier_normal_

→Tensor

xavier_normal_(tensor: Tensor, gain: float = 1.0)

source

Initialise tensor in-place with Xavier (Glorot) normal initialisation.

Gaussian counterpart of xavier_uniform_. Draws each entry from $\mathcal{N}(0, \sigma^2)$ with a standard deviation chosen to preserve activation variance through a stack of linear / mildly-nonlinear layers, as proposed in Glorot & Bengio (2010). Prefer this for tanh / sigmoid networks; use kaiming_normal_ for ReLU networks.

Parameters

tensorTensor

Tensor to initialise in place; must have at least 2 dimensions.

gainfloat= 1.0

Multiplicative gain factor — see calculate_gain. Default 1.0.

Returns

Tensor

tensor (mutated) for chaining.

Notes

The standard deviation is

\sigma = \text{gain} \cdot \sqrt{\frac{2}{n_\text{in} + n_\text{out}}},

which yields variance

\mathrm{Var}(W) = \frac{2 \cdot \text{gain}^2}{n_\text{in} + n_\text{out}}.

Examples

>>> import lucid
>>> from lucid.nn.init import xavier_normal_, calculate_gain
>>> w = lucid.empty(64, 32)
>>> xavier_normal_(w, gain=calculate_gain('tanh'))