relu

→Tensor

relu(x: Tensor, inplace: bool = False)

source

Apply the rectified linear unit function element-wise.

The simplest and most widely-used activation function in modern deep learning. Zeros out negative inputs and passes positive inputs through unchanged — a piecewise-linear sparsifier that gives neural networks their nonlinearity without saturating gradients for large positive activations.

Parameters

xTensor

Input tensor of any shape.

inplacebool= False

Reserved for API compatibility — currently a no-op (Lucid always allocates a fresh output).

Returns

Tensor

Element-wise ReLU output, same shape and dtype as x.

Notes

Mathematical definition:

\text{ReLU}(x) = \max(0,\, x)

Gradient is the indicator $\partial \text{ReLU}/\partial x = \mathbb{1}_{x > 0}$ — zero for negative inputs, one for positive, undefined at exactly zero (convention: 0). This causes the dying ReLU problem: a neuron that always produces negative pre-activations receives no gradient and stops learning. Variants like leaky_relu, elu, gelu, and silu address this by allowing a small (or smooth) gradient for negative inputs.

Despite its simplicity, ReLU remains a strong default for CNN backbones; transformers commonly prefer gelu for its smooth non-monotonic shape.

Examples

>>> import lucid
>>> from lucid.nn.functional import relu
>>> relu(lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0]))
Tensor([0., 0., 0., 1., 2.])