class

ReLU

extendsModule

ReLU(inplace: bool = False)

source edit

Rectified Linear Unit activation function.

Applies element-wise:

\text{ReLU}(x) = \max(0,\, x)

The simplest non-linear activation — sets all negative values to zero. Commonly used in hidden layers of deep networks because it avoids the vanishing-gradient problem that affects sigmoid and tanh for large inputs, and is inexpensive to compute.

Parameters

inplacebool= False

If True, the operation modifies the input tensor in-place, saving memory allocation. Default: False.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.ReLU()
>>> x = lucid.tensor([-1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([0., 0., 1., 2.])
>>> # Applied to a batch of feature vectors; negatives zeroed, shape preserved
>>> m = nn.ReLU(inplace=True)
>>> x = lucid.randn(4, 64)
>>> out = m(x)
>>> out.shape
(4, 64)

Used by 1

lucid.nn.modules

Constructors

dunder

init

→None

__init__(inplace: bool = False)

source edit

Initialise the ReLU module. See the class docstring for parameter semantics.

Instance methods

extra_repr

→str

extra_repr()

source edit

Return a string representation of the layer's configuration.

forward

→Tensor

forward(x: Tensor)

source edit

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.

>>> import lucid >>> import lucid.nn as nn >>> m = nn.ReLU() >>> x = lucid.tensor([-1.0, 0.0, 1.0, 2.0]) >>> m(x) tensor([0., 0., 1., 2.]) >>> # Applied to a batch of feature vectors; negatives zeroed, shape preserved >>> m = nn.ReLU(inplace=True) >>> x = lucid.randn(4, 64) >>> out = m(x) >>> out.shape (4, 64)