class

Tanhshrink

extendsModule

Tanhshrink()

source

Tanhshrink activation function.

Applies element-wise:

\text{Tanhshrink}(x) = x - \tanh(x)

Subtracts the tanh of the input from the input itself. Near zero the output is approximately $x^3/3$ (the tanh Taylor residual), so the function is smooth and cubic near the origin. For large inputs, $\tanh(x) \to \pm 1$ , so the output approaches $x \mp 1$ , behaving asymptotically like the identity.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

Tanhshrink is used as a soft thresholding operator in some sparse representation and compressed-sensing networks.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.Tanhshrink()
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([-1.0360, -0.2384,  0.    ,  0.2384,  1.0360])
>>> # Smooth sparse regularisation in an encoder
>>> x = lucid.randn(4, 256)
>>> out = m(x)
>>> out.shape
(4, 256)

Methods (1)

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.