silu

→Tensor

silu(x: Tensor, inplace: bool = False)

source

Sigmoid Linear Unit (a.k.a. Swish-1) activation.

Smooth, non-monotonic activation popularised by Ramachandran et al. (2017) as the result of a neural architecture search over activation functions. Equivalent to Swish with $\beta = 1$ .

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

inplacebool= False

Accepted for API compatibility; currently ignored.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{SiLU}(x) = x \, \sigma(x) = \frac{x}{1 + e^{-x}}

Has a small negative bump near $x \approx -1.28$ (minimum value $\approx -0.278$ ); approaches 0 from below for large negative input and approaches the identity for large positive input. The derivative is

\text{SiLU}'(x) = \sigma(x) + x\sigma(x)(1 - \sigma(x)),

which is smooth and unbounded above. Often outperforms ReLU in deep CNNs and is the default in EfficientNet and many MoE blocks.

Examples

>>> import lucid
>>> from lucid.nn.functional import silu
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> silu(x)
Tensor([-0.2384, -0.2689,  0.0000,  0.7311,  1.7616])