softsign

→Tensor

softsign(x: Tensor)

source

Softsign activation.

A bounded $(-1, 1)$ activation that saturates polynomially rather than exponentially — its tails are much heavier than tanh's, so gradients vanish more slowly in deep networks.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

Returns

Tensor

Activated tensor with the same shape as x, values in $(-1, 1)$ .

Notes

\text{softsign}(x) = \frac{x}{1 + |x|}, \qquad \text{softsign}'(x) = \frac{1}{(1 + |x|)^2}

Derivative decays like $1/x^2$ rather than $\tanh$ 's $1/\cosh^2$ , so saturation is gentler. Lacks the bi-Lipschitz smoothness of tanh near zero (kink at $x = 0$ from the absolute value) but is monotone and zero-centred.

Examples

>>> import lucid
>>> from lucid.nn.functional import softsign
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> softsign(x)
Tensor([-0.6667, -0.5000,  0.0000,  0.5000,  0.6667])