celu

→Tensor

celu(x: Tensor, alpha: float = 1.0, inplace: bool = False)

source

Continuously-differentiable Exponential Linear Unit (Barron 2017).

A reparameterisation of elu whose first derivative is continuous at the origin for every $\alpha > 0$ , not only the specific value $\alpha = 1$ . This makes CELU friendlier to optimisers that exploit smooth gradients.

Parameters

xTensor

Input tensor of any shape; activation is element-wise.

alphafloat= 1.0

Saturation value

\alpha > 0

for negative inputs. Default 1.0.

inplacebool= False

Accepted for API compatibility; currently ignored.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{CELU}(x) = \max(0, x) + \min\!\big(0, \alpha (e^{x/\alpha} - 1)\big)

At the origin both branches have derivative $1$ regardless of $\alpha$ , removing the kink ELU has when $\alpha \ne 1$ . Reduces to ELU when $\alpha = 1$ .

Examples

>>> import lucid
>>> from lucid.nn.functional import celu
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0])
>>> celu(x, alpha=1.0)
Tensor([-0.8647, -0.6321,  0.0000,  1.0000])