leaky_relu

→Tensor

leaky_relu(x: Tensor, negative_slope: float = 0.01, inplace: bool = False)

source

Leaky rectified linear unit activation.

A simple modification of relu that lets a small, non-zero gradient pass through for negative inputs. This avoids the dying ReLU problem where neurons stuck on the negative side stop updating because their gradient is exactly zero.

Parameters

xTensor

Input tensor of any shape; activation is applied element-wise.

negative_slopefloat= 0.01

Slope

\alpha

of the linear branch for negative inputs. Default 0.01.

inplacebool= False

Accepted for API compatibility; currently ignored.

Returns

Tensor

Activated tensor with the same shape as x.

Notes

\text{LeakyReLU}(x) = \max(0, x) + \alpha \min(0, x)

The derivative is $1$ for $x > 0$ and $\alpha$ for $x \le 0$ . Unlike relu, the derivative is everywhere non-zero (assuming $\alpha \ne 0$ ), so gradient signal continues to flow through inactive units. Use Leaky ReLU when ReLU networks are showing many permanently dead neurons; for a learnable slope, see prelu.

Examples

>>> import lucid
>>> from lucid.nn.functional import leaky_relu
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0])
>>> leaky_relu(x, negative_slope=0.1)
Tensor([-0.2000, -0.1000,  0.0000,  1.0000])