tanhshrink

→Tensor

tanhshrink(x: Tensor)

source edit

Tanh shrinkage activation.

Behaves like a smooth high-pass filter: subtracts the tanh-saturated component of the input, leaving small linear residuals near zero and near-linear behaviour for large magnitudes.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{TanhShrink}(x) = x - \tanh(x)

For $|x| \ll 1$ , $\tanh(x) \approx x - x^3/3$ , so $\text{TanhShrink}(x) \approx x^3/3$ — small signals are cubically suppressed. For $|x| \gg 1$ , $\tanh$ saturates to $\pm 1$ so the output behaves like $x \mp 1$ . $C^\infty$ -smooth and monotonic.

Examples

>>> import lucid
>>> from lucid.nn.functional import tanhshrink
>>> x = lucid.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
>>> tanhshrink(x)
tensor([-1.036, -0.03788, 0., 0.03788, 1.036])

Used by 2