Tanh-approximation GeLU $y = 0.5\,x\,(1 + \tanh(\sqrt{2/\pi}\,(x + 0.044715\,x^3)))$.

See Also

gelu_exact_op : Exact erf-based variant. GeluBackward : Autograd node implementing the gradient rule.