xavier_normal

→Tensor

xavier_normal_(tensor: Tensor, gain: float = 1.0)

source edit

Initialise tensor in-place with Xavier (Glorot) normal initialisation.

Gaussian counterpart of xavier_uniform_. Draws each entry from $\mathcal{N}(0, \sigma^2)$ with a standard deviation chosen to preserve activation variance through a stack of linear / mildly-nonlinear layers, as proposed in Glorot & Bengio (2010). Prefer this for tanh / sigmoid networks; use kaiming_normal_ for ReLU networks.

Parameters

tensorTensor

Tensor to initialise in place; must have at least 2 dimensions.

gainfloat= 1.0

Multiplicative gain factor — see calculate_gain. Default 1.0.

Returns

Tensor

tensor (mutated) for chaining.

Notes

The standard deviation is

\sigma = \text{gain} \cdot \sqrt{\frac{2}{n_\text{in} + n_\text{out}}},

which yields variance

\mathrm{Var}(W) = \frac{2 \cdot \text{gain}^2}{n_\text{in} + n_\text{out}}.

Examples

>>> import lucid
>>> from lucid.nn.init import xavier_normal_, calculate_gain
>>> w = lucid.empty(64, 32)
>>> lucid.manual_seed(0)
>>> _ = xavier_normal_(w, gain=calculate_gain('tanh'))
>>> std = calculate_gain('tanh') * (2 / (32 + 64)) ** 0.5
>>> abs(w.std().item() - std) < 0.1 * std
True

>>> import lucid >>> from lucid.nn.init import xavier_normal_, calculate_gain >>> w = lucid.empty(64, 32) >>> lucid.manual_seed(0) >>> _ = xavier_normal_(w, gain=calculate_gain('tanh')) >>> std = calculate_gain('tanh') * (2 / (32 + 64)) ** 0.5 >>> abs(w.std().item() - std) < 0.1 * std True