loss.backward(), End to End
SequenceWhat loss.backward() actually runs: the ones seed, the reverse-DFS walk, version checks and gradient accumulation.
8 participants · 14 messages · 4 guided chapters
Key facts
How the order is chosen
- topo_order is the reverse of an iterative DFS post-order over next_edges
- No dependency counters, no ready queue: one single-threaded pass
Version check
- forward stamps input versions with set_saved_versions
- An in-place change to a saved input raises VersionMismatch
Releasing the graph
- retain_graph=False frees saved tensors node by node
- create_graph=True forces retain_graph and takes backward_for_graph
Guided chapters
- 01
Seed and entry
Tensor.backward is a thin wrapper; the real work is one C++ Engine::backward call.
- 02
Reverse walk
Each node runs once in topo_order, and grads flow along next_edges.
- 03
Version check
If a saved tensor was modified in place after forward, the pass stops here.
- 04
Python hooks
Tensor hooks run in Python after the engine returns.