loss.backward(), End to End

loss.backward(), End to End A sequence diagram generated by Archify. loss.backward() engine_backward With a gradient argument it differentiates (self * g).sum() instead, so only a scalar seed goes in empty seed Called with the GIL held. ones seed → fusion pass → topo_order (reverse DFS post-order) apply(1) Per node: validate_versions → apply → release_saved g grad broadcast back to the input shape validate_versions Throws VersionMismatch if a saved input changed since forward apply(g) g·w, g·a reduce_grad_to_shape undoes broadcasting g·a Grads bound for the same node are summed in the pending map first w.grad += The first write moves the grad in; later ones use accumulate_into graph released With retain_graph=False the saved tensors are already freed; root.clear_grad_fn drops the chain return run hooks _dispatch_tensor_grad_hooks: a returned tensor replaces only that tensor's .grad .grad ready Entry Reverse walk Cleanup User · (a * w).sum() · Sequence participant User (a * w).sum() Tensor · .backward() · Sequence participant Tensor .backward() hooks · tensor hooks · Sequence participant hooks tensor hooks pybind11 · engine_backward · Sequence participant pybind11 engine_backward Engine · Engine::backward · Sequence participant Engine Engine::backward Sum node · SumBackward · Sequence participant Sum node SumBackward Mul node · MulBackward · Sequence participant Mul node MulBackward Leaf w · AccumulateGrad · Sequence participant Leaf w AccumulateGrad Legend request return version check runs after engine default message

How the order is chosen

  • • topo_order is the reverse of an iterative DFS post-order over next_edges
  • • No dependency counters, no ready queue: one single-threaded pass

Version check

  • • forward stamps input versions with set_saved_versions
  • • An in-place change to a saved input raises VersionMismatch

Releasing the graph

  • • retain_graph=False frees saved tensors node by node
  • • create_graph=True forces retain_graph and takes backward_for_graph