rms_norm_forward_f32

void rms_norm_forward_f32(const float * x, const float * gamma, float * y, float * saved_rstd, int outer, int N, double eps)

source edit

Single-precision RMSNorm forward — divides each row by its root-mean-square magnitude and scales by a per-feature gain.

Unlike LayerNorm there is no mean subtraction and no bias parameter, so the kernel only stores rstd = 1/sqrt(mean(x^2) + eps) per row. This makes RMSNorm faster and is the form used in many modern transformer variants.

Math

$y_i = \gamma_i \frac{x_i}{\sqrt{\frac{1}{N}\sum_j x_j^2 + \varepsilon}}.$

References

Zhang & Sennrich, "Root Mean Square Layer Normalization" (NeurIPS 2019).

Parameters

xconst float*

Input buffer of shape (outer, N).

gammaconst float*

Per-feature scale of shape (N,).

yfloat*

Output buffer of shape (outer, N).

saved_rstdfloat*

Per-row reciprocal RMS, shape (outer,); consumed by rms_norm_backward_f32.

outer, Nstd::size_t

Number of rows and normalised-axis length.

epsdouble

Stabiliser added to mean(x^2) before the square root; cast to float.