A Tensor's Autograd Life

A Tensor's Autograd Life A lifecycle diagram generated by Archify. 01 / Main path 02 / Graph kept + Recovery loop 03 / Not recorded Leaf tensor · requires_grad=True · Main path · AccumulateGrad 01 Leaf tensor requires_grad=True AccumulateGrad Graph recorded · grad_fn · saved_versions · Main path · forward 02 Graph recorded grad_fn · saved_versions forward Backward walk · topo_order · apply · Main path · Engine 03 Backward walk topo_order · apply Engine Free saved? · check retain_graph · Main path · per node 04 Free saved? check retain_graph per node .grad final · root grad_fn cleared · Main path · after hooks 05 .grad final root grad_fn cleared after hooks Graph kept · retain_graph=True · Graph kept · reusable Graph kept retain_graph=True reusable VersionMismatch · in-place after save · Recovery loop · recoverable VersionMismatch in-place after save recoverable No node · no_grad · inference_mode · Not recorded · GradMode off No node no_grad · inference_mode GradMode off retain_graph=True backward again version mismatch re-run forward GradMode off Legend start active state waiting decision terminal success failure / exit neutral

What gets recorded

  • • grad_fn (XxxBackward), next_edges and the input versions
  • • A leaf gets AccumulateGrad as its grad_fn on first use

Ways back

  • • VersionMismatch clears once forward re-runs and builds a new graph
  • • Only a graph kept with retain_graph=True can be walked again