class

Softplus

extendsModule

Softplus(beta: float = 1.0, threshold: float = 20.0)

source

Softplus activation function.

Applies element-wise:

\text{Softplus}(x) = \frac{1}{\beta} \ln\!\bigl(1 + e^{\beta x}\bigr)

For numerical stability the output falls back to the identity when $\beta x > \text{threshold}$ , avoiding overflow in the exponential. Softplus is a smooth, everywhere-differentiable approximation of ReLU, and is useful as the positivity-enforcing transformation in probabilistic models (e.g. predicting variances or scale parameters).

Parameters

betafloat= 1.0

Sharpness parameter

\beta

controlling how closely the curve approximates a hard hinge. Larger values approach ReLU. Default: 1.0.

thresholdfloat= 20.0

Above this value of

\beta x

the function falls back to the identity to prevent overflow. Default: 20.0.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

Setting beta=1 recovers the standard log-sum-exp formulation. For very large beta the function becomes numerically equivalent to ReLU almost everywhere.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.Softplus()
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([0.1269, 0.3133, 0.6931, 1.3133, 2.1269])
>>> # Sharper approximation with beta=5
>>> m_sharp = nn.Softplus(beta=5.0)
>>> x = lucid.randn(3, 64)
>>> out = m_sharp(x)
>>> out.shape
(3, 64)

Methods (3)

dunder

init

→None

__init__(beta: float = 1.0, threshold: float = 20.0)

source

Initialise the Softplus module. See the class docstring for parameter semantics.

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.

extra_repr

→str

extra_repr()

source

Return a string representation of the layer's configuration.