GeluBackward

Autograd node for the tanh-approximation GeLU $y = 0.5\,x\,(1 + \tanh(\sqrt{2/\pi}\,(x + 0.044715\,x^3)))$.

The approximate form (Hendrycks & Gimpel 2016, eq. 3) is used by default when callers do not request the exact erf-based variant; it is the form implemented by F.gelu(x, approximate="tanh") in the Python wrapper. The backward is delegated to the backend because the analytic derivative involves both tanh and sech^2 over a cubic polynomial of $x$, and fusing them in a single kernel is materially faster than chaining storage primitives.

Math

$y = \tfrac{x}{2}\,\bigl(1 + \tanh\bigl(\sqrt{2/\pi}\,(x + 0.044715\,x^3)\bigr)\bigr)$

References

Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)", arXiv:1606.08415.