hardswish

→Tensor

hardswish(x: Tensor)

source

Hard Swish activation — piecewise linear approximation of silu.

Introduced in MobileNetV3 (Howard et al. 2019) as a compute-friendly replacement for the sigmoid in Swish: the exponential is approximated by a clipped linear function, which is far cheaper on mobile / quantised hardware while preserving the shape of the activation.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{HardSwish}(x) = x \cdot \frac{\text{ReLU6}(x + 3)}{6} = \begin{cases} 0 & x \le -3 \\ x(x+3)/6 & -3 < x < 3 \\ x & x \ge 3 \end{cases}

Continuous but only piecewise- $C^1$ (the second derivative jumps at $x = \pm 3$ ). Cheap to evaluate and to quantise — preferred over silu in mobile architectures.

Examples

>>> import lucid
>>> from lucid.nn.functional import hardswish
>>> x = lucid.tensor([-4.0, -1.0, 0.0, 1.0, 4.0])
>>> hardswish(x)
Tensor([ 0.0000, -0.3333,  0.0000,  0.6667,  4.0000])