class

ReluBackward

ReluBackward

source edit

Python wrappers

Autograd node for the element-wise rectified linear unit $y = \max(0, x)$ .

Saves the input so the backward pass can rebuild the positive-input mask $\mathbb{1}\{x > 0\}$ and gate the upstream gradient through it. At the non-differentiable point $x = 0$ the subgradient is taken to be $0$ , which matches the reference framework's convention.

Math

$y = \max(0, x), \qquad \frac{\partial y}{\partial x} = \mathbb{1}\{x > 0\}$

Attributes

schema_v1OpSchema

"relu", AmpPolicy::KeepInput (ReLU is well-defined on integer dtypes, so the AMP layer leaves the input dtype untouched).

Notes

MLX exposes a native maximum(x, 0) kernel on the GPU stream; the CPU path goes through vDSP_vmax against a zero buffer. grad_formula_impl (used by create_graph=True) builds the mask as sign(relu(x)) so the inverse-mask multiplication stays inside the autograd graph for second-order differentiation.

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::relu to compute $y = \max(0, x)$ .

Methods

grad_formula

→Storage

int grad_formula(const int & g)

source edit

Eager-mode backward: $\partial L/\partial x = \mathbb{1}\{x > 0\} \cdot \partial L/\partial y$ , computed via a positive-input mask multiplied element-wise against the upstream gradient.

Parameters

gStorage

Upstream gradient dL/dy matching the forward output shape.

Returns

Storage

dL/dx at the input shape and dtype.

grad_formula_impl

int grad_formula_impl(const int & g, const int & x, const int &)

source edit

Graph-mode backward used when create_graph=True. Builds the mask as sign(relu(x)) (zero for $x \le 0$ , one for $x > 0$ ) so the operation is itself differentiable for higher-order gradients.

class

ReluBackward

ReluBackward

source edit

Python wrappers

Autograd node for the element-wise rectified linear unit $y = \max(0, x)$ .

Math

$y = \max(0, x), \qquad \frac{\partial y}{\partial x} = \mathbb{1}\{x > 0\}$

Attributes

schema_v1OpSchema

"relu", AmpPolicy::KeepInput (ReLU is well-defined on integer dtypes, so the AMP layer leaves the input dtype untouched).

Notes

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::relu to compute $y = \max(0, x)$ .

Methods

grad_formula

→Storage

int grad_formula(const int & g)

source edit

Eager-mode backward: $\partial L/\partial x = \mathbb{1}\{x > 0\} \cdot \partial L/\partial y$ , computed via a positive-input mask multiplied element-wise against the upstream gradient.

Parameters

gStorage

Upstream gradient dL/dy matching the forward output shape.

Returns

Storage

dL/dx at the input shape and dtype.

grad_formula_impl

int grad_formula_impl(const int & g, const int & x, const int &)

source edit

Graph-mode backward used when create_graph=True. Builds the mask as sign(relu(x)) (zero for $x \le 0$ , one for $x > 0$ ) so the operation is itself differentiable for higher-order gradients.