softplus

→Tensor

softplus(x: Tensor, beta: float = 1.0, threshold: float = 20.0)

source

Smooth approximation of relu.

Softplus is a strictly positive, $C^\infty$ smooth function that behaves like $x$ for large positive inputs and like $e^x$ for large negative inputs. Its derivative is sigmoid, which is why softplus appears naturally as the log-partition of a Bernoulli and as the analytic primitive of the logistic function.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

betafloat= 1.0

Sharpness parameter

\beta > 0

. Larger beta makes the curve approach a hard ReLU. Default 1.0.

thresholdfloat= 20.0

For numerical stability the formula collapses to the identity

x

whenever

\beta x > \text{threshold}

(the

e^{\beta x}

term would otherwise overflow). Default 20.0.

Returns

Tensor

Activated tensor with the same shape as x, strictly positive.

Notes

\text{softplus}_\beta(x) = \frac{1}{\beta}\log\!\big(1 + e^{\beta x}\big)

Derivative is $\sigma(\beta x)$ . The overflow guard at threshold exploits that for large $\beta x$ the function is indistinguishable from $x$ in finite precision. The engine's bare softplus kernel implements the special case $\beta = 1$ , $\text{threshold} \to \infty$ ; when either argument deviates the full formula is composed so autograd still flows through the existing backward nodes.

Examples

>>> import lucid
>>> from lucid.nn.functional import softplus
>>> x = lucid.tensor([-2.0, 0.0, 2.0])
>>> softplus(x)
Tensor([0.1269, 0.6931, 2.1269])