class

LeakyReLU

extendsModule

LeakyReLU(negative_slope: float = 0.01, inplace: bool = False)

source

Leaky Rectified Linear Unit activation function.

Applies element-wise:

\text{LeakyReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ \alpha \cdot x & \text{otherwise} \end{cases}

Unlike standard ReLU, neurons with negative pre-activations receive a small gradient $\alpha$ during back-propagation, avoiding the "dying ReLU" phenomenon where units become permanently inactive.

Parameters

negative_slopefloat= 0.01

Slope

\alpha

applied to negative inputs. Default: 0.01.

inplacebool= False

If True, modifies the input tensor in-place. Default: False.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.LeakyReLU(negative_slope=0.1)
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([-0.2, -0.1,  0. ,  1. ,  2. ])
>>> # Default slope 0.01 — very small leak
>>> m = nn.LeakyReLU()
>>> x = lucid.randn(3, 32)
>>> out = m(x)
>>> out.shape
(3, 32)

Methods (3)

dunder

init

→None

__init__(negative_slope: float = 0.01, inplace: bool = False)

source

Initialise the LeakyReLU module. See the class docstring for parameter semantics.

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.

extra_repr

→str

extra_repr()

source

Return a string representation of the layer's configuration.