Terminal backward node that accumulates an incoming gradient into a leaf tensor's .grad storage.
An AccumulateGrad is attached to a leaf TensorImpl's
grad_fn slot the first time the tensor is used in a computation that
requires gradients (see _register_python_backward_node and the op
builders under ops/). Leaves do not themselves produce a grad_fn
during their own "forward" step, so this sentinel terminates the backward
walk: its next_edges() is always empty.
Apply Semantics
On each call to apply(grad_out):
- If the leaf has been garbage-collected (
weak_ptrexpired), the gradient is silently discarded. - If the leaf no longer requires gradients (e.g.
requires_grad_(False)was called between forward and backward), the gradient is discarded. - Incoming gradients are cast to the leaf's own dtype to handle AMP / autocast paths where the same parameter may be reached via different effective dtypes (e.g. F16 from a Conv and F32 from a ForceFP32 branch).
- If the leaf has no gradient yet,
grad_outis moved in directly — avoiding the cost of allocating a zero buffer and adding to it. Otherwiseaccumulate_into()performs an in-place+=using the appropriate backend (Accelerate on CPU, MLX on GPU).
Ownership
AccumulateGrad holds only a weak_ptr to its leaf so that it does
not artificially extend the tensor's lifetime while the backward graph is
alive. Non-leaf intermediate tensors never carry an AccumulateGrad —
their gradient flows through whatever produced them and is only stored
transiently in the engine's pending map (or in .grad if the user
explicitly called retain_grad on the intermediate).
See Also
Engine : reverse-mode driver that dispatches into this node.
Node : abstract base class.
Constructors
1AccumulateGrad
void AccumulateGrad(int leaf)Construct an AccumulateGrad bound to a specific leaf tensor.
Parameters
leafstd::weak_ptr<TensorImpl>.grad slot this node writes into. Must satisfy leaf->is_leaf() == true; not asserted here but enforced at all call sites (op builders, hook-edge constructors).Methods
4Write grad_out into the leaf's gradient storage.
The first call moves the gradient in directly; subsequent calls
accumulate += in place via the backend-appropriate kernel.
Parameters
grad_outStorageReturns
std::vector<Storage>Always empty — AccumulateGrad has no outgoing edges and the engine has no further work to schedule for this branch.
Graph-mode variant invoked when Engine::backward runs with create_graph=true. Stores grad_out (a TensorImpl that may carry its own grad_fn) into the leaf's grad_impl slot so the gradient tensor itself remains differentiable for higher-order gradient computations.
Parameters
grad_outconst TensorImplPtr&Returns
std::vector<TensorImplPtr>Always empty for the same reason as apply.
int leaf()Accessor for the leaf weak_ptr.
Returns
std::weak_ptr<TensorImpl>A weak reference to the bound leaf — may be expired. Used by tests and graph-inspection utilities.
Human-readable node name surfaced in error messages and graph dumps.