class

LocalResponseNorm

extendsModule

LocalResponseNorm(size: int, alpha: float = 0.0001, beta: float = 0.75, k: float = 1.0)

source edit

Local Response Normalization (LRN) across adjacent channels.

Normalises each activation by a sum of squared activations in a local neighbourhood of size channels centred on the same spatial position:

y_c = \frac{x_c}{\left(k + \alpha \sum_{j=\max(0,\,c-\lfloor \text{size}/2\rfloor)}^{\min(C-1,\,c+\lfloor \text{size}/2\rfloor)} x_j^2\right)^{\beta}}

where $C$ is the total number of channels, and the window $[c - \lfloor \text{size}/2 \rfloor,\; c + \lfloor \text{size}/2 \rfloor]$ is clipped at the channel boundaries.

LRN was introduced in the AlexNet paper as a biologically-inspired lateral inhibition mechanism: strongly-activated neurons suppress their neighbors, encouraging competition across feature detectors at the same spatial location. It has largely been superseded by Batch Normalization in modern architectures, but remains available for reproducing historical results.

Parameters

sizeint

Number of adjacent channels included in the normalization window (the

\text{size}

term in the formula above). Must be a positive integer.

alphafloat= 0.0001

Scaling factor

\alpha

for the squared sum. Default: 1e-4.

betafloat= 0.75

Exponent

\beta

applied to the normalization term. Default: 0.75.

kfloat= 1.0

Bias constant

k

that prevents division by zero. Default: 1.0.

Notes

Input: $(N, C, *)$ where $*$ denotes zero or more spatial dimensions. The channel axis must be axis 1.
Output: same shape as the input.

The implementation uses a sliding-window sum over the squared input, applied via unfold_dim on the padded channel axis. This avoids an explicit Python loop over channels and is fully differentiable through the Lucid engine.
Inputs with fewer than 2 dimensions are returned unchanged.
The original AlexNet used size=5, alpha=1e-4, beta=0.75, k=2.0 — adjusting k is important to control how strongly unsuppressed channels are penalized.

Examples

Reproduce the AlexNet LRN configuration:
>>> import lucid
>>> import lucid.nn as nn
>>> lrn = nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2.0)
>>> x = lucid.randn(2, 96, 27, 27)   # e.g. after first conv in AlexNet
>>> out = lrn(x)
>>> out.shape
(2, 96, 27, 27)
Minimal example with a 1-D channel tensor:
>>> lrn_small = nn.LocalResponseNorm(size=3)
>>> x2 = lucid.randn(4, 32, 64)   # (N, C, L)
>>> out2 = lrn_small(x2)
>>> out2.shape
(4, 32, 64)

Used by 1

lucid.nn.modules

Constructors

dunder

init

→None

__init__(size: int, alpha: float = 0.0001, beta: float = 0.75, k: float = 1.0)

source edit

Initialise the LocalResponseNorm module. See the class docstring for parameter semantics.

Instance methods

extra_repr

→str

extra_repr()

source edit

Return a string representation of the layer's configuration.

forward

→Tensor

forward(x: Tensor)

source edit

Apply normalisation to the input tensor.

Parameters

inputTensor

Input tensor whose shape is documented in the class docstring.

Returns

Tensor

Normalised tensor of the same shape as input.

Reproduce the AlexNet LRN configuration: >>> import lucid >>> import lucid.nn as nn >>> lrn = nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75, k=2.0) >>> x = lucid.randn(2, 96, 27, 27) # e.g. after first conv in AlexNet >>> out = lrn(x) >>> out.shape (2, 96, 27, 27) Minimal example with a 1-D channel tensor: >>> lrn_small = nn.LocalResponseNorm(size=3) >>> x2 = lucid.randn(4, 32, 64) # (N, C, L) >>> out2 = lrn_small(x2) >>> out2.shape (4, 32, 64)