sparse_

→Tensor

sparse_(tensor: Tensor, sparsity: float, std: float = 0.01)

source

Initialise a 2-D tensor in-place with a sparse random matrix.

For each column, a random subset of floor(sparsity * rows) rows is set to zero; the remaining entries are drawn from $\mathcal{N}(0, \text{std}^2)$ . Sparse initialisation, due to Martens (2010), encourages each output unit to depend on only a small number of inputs at the start of training and helps in ill-conditioned regimes — particularly for very wide layers where a dense Gaussian would have an unfavourable condition number.

Parameters

tensorTensor

Two-dimensional tensor to fill in place.

sparsityfloat

Fraction of entries in each column that are zeroed. Must lie in [0, 1].

stdfloat= 0.01

Standard deviation of the Gaussian used for the non-zero entries. Default 0.01.

Returns

Tensor

tensor (mutated) for chaining.

Raises

ValueError

If tensor.ndim != 2 or sparsity is outside [0, 1].

Notes

The expected per-entry variance is

\mathrm{Var}(W_{ij}) = (1 - \text{sparsity}) \cdot \text{std}^2.

Examples

>>> import lucid
>>> from lucid.nn.init import sparse_
>>> w = lucid.empty(256, 256)
>>> sparse_(w, sparsity=0.9, std=0.01)