class

PReLU

extendsModule

PReLU(num_parameters: int = 1, init: float = 0.25, device: DeviceLike = None, dtype: DTypeLike = None)

source

Parametric Rectified Linear Unit activation function.

Applies element-wise:

\text{PReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ \alpha \cdot x & \text{otherwise} \end{cases}

Unlike LeakyReLU, the slope $\alpha$ is a learnable parameter updated during back-propagation. A single shared slope can be used for all channels (num_parameters=1) or each channel can have its own slope (num_parameters=num_channels).

Parameters

num_parametersint= 1

Number of learnable slopes. Use 1 for a single shared slope or set to the number of input channels for per-channel slopes. Default: 1.

initfloat= 0.25

Initial value for all slope parameters. Default: 0.25.

deviceDeviceLike= None

Device on which the parameter tensor is allocated. Default: None (uses the default device).

dtypeDTypeLike= None

Data type of the parameter tensor. Default: None (uses the default floating-point type).

Attributes

weightParameter of shape ``(num_parameters,)``

Learnable negative slopes

\alpha

. Updated by the optimiser during training.

Notes

Input: $(*)$ — any shape.
Output: $(*)$ — same shape as input.

When num_parameters > 1, the input is expected to have the channel dimension second (i.e. shape $(N, C, *)$ ), and num_parameters must equal $C$ .

Examples

>>> import lucid
>>> import lucid.nn as nn
>>> m = nn.PReLU(num_parameters=1, init=0.25)
>>> x = lucid.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
>>> m(x)
tensor([-0.5, -0.25,  0.  ,  1.  ,  2.  ])
>>> # Per-channel slopes for a feature map with 64 channels
>>> m = nn.PReLU(num_parameters=64)
>>> x = lucid.randn(8, 64, 16, 16)
>>> out = m(x)
>>> out.shape
(8, 64, 16, 16)

Methods (3)

dunder

init

→None

__init__(num_parameters: int = 1, init: float = 0.25, device: DeviceLike = None, dtype: DTypeLike = None)

source

Initialise the PReLU module. See the class docstring for parameter semantics.

forward

→Tensor

forward(x: Tensor)

source

Apply the activation function element-wise.

Parameters

inputTensor

Input tensor of arbitrary shape.

Returns

Tensor

Output tensor of the same shape as input.

extra_repr

→str

extra_repr()

source

Return a string representation of the layer's configuration.