Reverse-mode autograd engine — drives the backward pass over a recorded computation graph rooted at a single output TensorImpl.
Engine is a stateless class that exposes a single static entry point
(Engine::backward). It is never instantiated; all per-call state
(worklist, accumulated gradients, visited set) lives on the stack of the
invoking thread.
Algorithm
- Compute a reverse-topological ordering of the backward graph using an iterative post-order DFS (the recursive form would overflow on deep networks).
- Optionally run a fusion pass that collapses adjacent backward nodes
(e.g.
LinearBackward+ReluBackward) into a single fused node. - Walk the ordering once, calling each node's
apply(); gradients arriving at a node from multiple producers are summed into a pending map before the node is executed. - Leaves (tensors with no
grad_fn) are reached through theirAccumulateGradsentinel, which writes the final gradient intoleaf.grad.
Thread Safety
Two concurrent backward calls on the same graph are undefined
behaviour — nodes are consumed and (when retain_graph=false) destroyed
exactly once. Independent graphs may be driven from separate threads.
See Also
AccumulateGrad : terminal node that writes into leaf.grad.
Node : abstract base for every backward graph node.
Static methods
1backward
void backward(const int & root, int grad_seed, bool retain_graph, bool create_graph)Run reverse-mode automatic differentiation starting from root.
Walks the computation graph attached to root->grad_fn() in reverse
topological order, computes per-edge input gradients via each node's
apply() method, accumulates contributions at branch points, and
hands terminal gradients to AccumulateGrad for in-place
accumulation into leaf.grad.
Parameters
rootstd::shared_ptr<TensorImpl>root has no grad_fn it is treated as a leaf and the seed is accumulated directly into root->grad.grad_seedStorage= Noneroot. An empty Storage (default) is replaced by a ones-tensor of root's shape / dtype / device — the common case of differentiating a scalar loss.retain_graphbool= Nonefalse (default), each node's release_saved() is called immediately after its apply() and root->grad_fn is cleared on return, so a second backward() call is impossible. Pass true to preserve the graph for multiple backward calls.create_graphbool= Nonetrue, the backward pass itself is recorded in the autograd graph so that higher-order gradients can be taken on the resulting .grad tensors. Implies retain_graph=true because the forward nodes are re-used by the new graph. Concrete nodes must override apply_for_graph for this mode; nodes lacking graph support raise a clear error naming the op.Raises
:runtime_errorroot is null, if a node returns an input_grads vector whose size disagrees with its outgoing edges (and both are non-empty), or if validate_versions() detects an in-place mutation of a saved input tensor.Notes
The engine consumes nodes destructively when retain_graph=false:
release_saved() frees the forward tensors each node had stashed
for its backward formula, and clear_grad_fn() on root
severs the producer→graph reference so the chain of shared_ptrs
collapses.
Examples
>>> # Pseudo-C++: differentiate a scalar loss
>>> Engine::backward(loss);
>>> // Now every leaf with requires_grad=true has its .grad populated.