Implementing kernel
C++ engine symbols that back this Python API.Softplus activation function.
Applies element-wise:
For numerical stability the output falls back to the identity when , avoiding overflow in the exponential. Softplus is a smooth, everywhere-differentiable approximation of ReLU, and is useful as the positivity-enforcing transformation in probabilistic models (e.g. predicting variances or scale parameters).
Parameters
betafloat= 1.0Sharpness parameter controlling how closely the curve
approximates a hard hinge. Larger values approach ReLU.
Default:
1.0.thresholdfloat= 20.0Above this value of the function falls back to the
identity to prevent overflow. Default:
20.0.Notes
- Input: — any shape.
- Output: — same shape as input.
Setting beta=1 recovers the standard log-sum-exp formulation.
For very large beta the function becomes numerically equivalent to
ReLU almost everywhere.
Examples
>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.Softplus()
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([0.1269, 0.3133, 0.6931, 1.3133, 2.1269])
>>> # Sharper approximation with beta=5
>>> m_sharp = nn.Softplus(beta=5.0)
>>> x = lucid.randn(3, 64)
>>> out = m_sharp(x)
>>> out.shape
(3, 64)