Reverse-mode autograd engine — drives the backward pass over a recorded computation graph rooted at a single output TensorImpl.
Engine is a stateless class that exposes a single static entry point
(Engine::backward). It is never instantiated; all per-call state
(worklist, accumulated gradients, visited set) lives on the stack of the
invoking thread.
Algorithm
- Compute a reverse-topological ordering of the backward graph using an iterative post-order DFS (the recursive form would overflow on deep networks).
- Optionally run a fusion pass that collapses adjacent backward nodes
(e.g.
LinearBackward+ReluBackward) into a single fused node. - Walk the ordering once, calling each node's
apply(); gradients arriving at a node from multiple producers are summed into a pending map before the node is executed. - Leaves (tensors with no
grad_fn) are reached through theirAccumulateGradsentinel, which writes the final gradient intoleaf.grad.
Thread Safety
Two concurrent backward calls on the same graph are undefined
behaviour — nodes are consumed and (when retain_graph=false) destroyed
exactly once. Independent graphs may be driven from separate threads.
See Also
AccumulateGrad : terminal node that writes into leaf.grad.
Node : abstract base for every backward graph node.
Static methods
2backward
void backward(const int & root, int grad_seed, bool retain_graph, bool create_graph)Run reverse-mode automatic differentiation starting from root.
Walks the computation graph attached to root->grad_fn() in reverse
topological order, computes per-edge input gradients via each node's
apply() method, accumulates contributions at branch points, and
hands terminal gradients to AccumulateGrad for in-place
accumulation into leaf.grad.
Parameters
rootstd::shared_ptr<TensorImpl>root has no grad_fn it is treated as a leaf and the seed is accumulated directly into root->grad.grad_seedStorage= Noneroot. An empty Storage (default) is replaced by a ones-tensor of root's shape / dtype / device — the common case of differentiating a scalar loss.retain_graphbool= Nonefalse (default), each node's release_saved() is called immediately after its apply() and root->grad_fn is cleared on return, so a second backward() call is impossible. Pass true to preserve the graph for multiple backward calls.create_graphbool= Nonetrue, the backward pass itself is recorded in the autograd graph so that higher-order gradients can be taken on the resulting .grad tensors. Implies retain_graph=true because the forward nodes are re-used by the new graph. Concrete nodes must override apply_for_graph for this mode; nodes lacking graph support raise a clear error naming the op.Raises
:runtime_errorroot is null, if a node returns an input_grads vector whose size disagrees with its outgoing edges (and both are non-empty), or if validate_versions() detects an in-place mutation of a saved input tensor.Notes
The engine consumes nodes destructively when retain_graph=false:
release_saved() frees the forward tensors each node had stashed
for its backward formula, and clear_grad_fn() on root
severs the producer→graph reference so the chain of shared_ptrs
collapses.
Examples
>>> # Pseudo-C++: differentiate a scalar loss
>>> Engine::backward(loss);
>>> // Now every leaf with requires_grad=true has its .grad populated.Differentiate root with respect to inputs without writing any .grad.
The functional counterpart to backward. It walks the same
graph, but instead of letting gradients terminate in the
AccumulateGrad nodes that own each leaf's .grad slot, it
intercepts them and returns them to the caller. No tensor's gradient
state is read or modified — not the requested inputs', and not any
other leaf's.
That last part is the reason this exists. Emulating it in Python by
running a full backward and then restoring .grad can only
restore the tensors the caller named; every other leaf in the graph
keeps whatever the traversal deposited, which silently corrupts
parameters the caller never mentioned.
See Also
Engine::backward — the accumulating counterpart.
Parameters
rootconst std::shared_ptr<TensorImpl>&grad_seedStorageroot. An empty storage means "ones", matching backward.inputsconst std::vector<std::shared_ptr<TensorImpl>>&retain_graphbool, default= falsecreate_graphbool, default= true for the returned gradients to beretain_graph.Returns
std::vector<TensorImplPtr>One entry per requested input, in order. An entry is null when that input lies outside root's graph — the caller decides whether that is an error.
Raises
LucidErrorroot or any entry of inputs is null.Notes
A leaf is matched by the AccumulateGrad node bound to it; an
interior tensor is matched by its own grad_fn. Interior captures
do not stop the traversal, so asking for a tensor halfway down the
graph still lets gradients reach everything below it.