Autograd node for Global Response Normalization (GRN), as used in ConvNeXt V2.
For 4-D input of shape (N, C, H, W), GRN first computes the per
(n, c) L2 norm over the spatial axes, then normalises each
channel's norm by the mean across channels, and finally applies a
learned per-channel affine that modulates the input rather than
replacing it:
(The trailing residual matches the ConvNeXt-V2 paper; the
affine is applied to the modulated signal rather than to
.) saved_Nx_ stores broadcast to the per-spatial
shape needed by the backward closed-form.
Inherits FuncOp<GlobalResponseNormBackward, 3> with three
saved-input slots — x, gamma, and beta.
References
Woo et al., "ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders" (CVPR 2023, arXiv:2301.00808).
Attributes
schema_v1OpSchemaname="global_response_norm", version=1, amp_policy=ForceFP32, produces_grad=true).eps_double1e-6.saved_Nx_Storage(N, C, 1, 1).Static methods
1forward
→TensorImplPtrint forward(const int & x, const int & gamma, const int & beta, double eps)Run the forward pass.
Computes the per (n, c) L2 spatial norm, normalises it by
the per-image mean across channels, modulates x by the
result, applies the per-channel affine + residual, caches
saved_Nx_, and (when grad mode is on) registers this as
the grad_fn of the returned output.
Parameters
xTensorImplPtr(N, C, H, W).gammaTensorImplPtr(C,).betaTensorImplPtr(C,).epsdoubleReturns
TensorImplPtrNormalised output, same shape as x.
Methods
1Compute gradients for the three saved inputs.
Returns {dx, d_gamma, d_beta} in the saved-input order
using the cached saved_Nx_.
Parameters
grad_outStorageReturns
std::vector<Storage>Three-element vector {dx, d_gamma, d_beta}.