class

Softsign

extendsModule

Softsign()

source

Softsign activation function.

Applies element-wise:

\text{Softsign}(x) = \frac{x}{1 + |x|}

Maps all real inputs to the open interval $(-1, 1)$ , similar to tanh but with polynomial rather than exponential saturation — meaning the function approaches its limits more slowly and gradients remain larger for moderately large inputs.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input, values in $(-1, 1)$ .

Softsign saturates more slowly than tanh, so it is less prone to vanishing gradients for large activations. It is sometimes used as a tanh substitute in recurrent networks.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.Softsign()
>>> x = lucid.tensor([-4.0, -1.0, 0.0, 1.0, 4.0])
>>> m(x)
tensor([-0.8   , -0.5   ,  0.    ,  0.5   ,  0.8   ])
>>> # Slower saturation compared to tanh
>>> x = lucid.randn(8, 64)
>>> out = m(x)
>>> out.shape
(8, 64)

Methods (1)

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.