mish

→Tensor

mish(x: Tensor)

source

Mish activation function.

Self-regularising non-monotonic activation introduced by Misra (2019). Like silu it has a small negative dip and approaches the identity for large positive inputs, but its tanh-of-softplus form gives slightly smoother gradients which tend to help convergence in CV tasks (YOLOv4 uses it throughout).

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{Mish}(x) = x \, \tanh\!\big(\text{softplus}(x)\big) = x \, \tanh\!\big(\ln(1 + e^x)\big)

$C^\infty$ -smooth, non-monotonic, unbounded above and bounded below (minimum $\approx -0.3088$ ). Empirically outperforms ReLU/Swish on some object-detection benchmarks at modest extra cost.

Examples

>>> import lucid
>>> from lucid.nn.functional import mish
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> mish(x)
Tensor([-0.2525, -0.3034,  0.0000,  0.8651,  1.9440])