a + b, End to End
SequenceOne a + b traced end to end — dtype promotion, SchemaGuard, vDSP or MLX, and the AddBackward node it leaves behind.
8 participants · 14 messages · 4 guided chapters
Key facts
One object, three roles
- AddBackward is the op, the BinaryKernel (CRTP) and the autograd node at once
- a + b skips the _ops registry: the injected __add__ calls the engine directly
CPU vs GPU
- CPU: vDSP_vadd computes on the spot
- GPU: mlx::core::add only adds a node; it runs at .item() or eval()
Where it is rejected
- DtypeMismatch, DeviceMismatch, ShapeMismatch
- C++ never promotes dtypes; Python aligns them first
Guided chapters
- 01
Python to C++
The injected __add__ finishes scalar coercion and dtype promotion, then hands off to pybind11.
- 02
Validation and schema
BinaryKernel checks dtype, device and shape; SchemaGuard settles determinism and the AMP dtype.
- 03
CPU and GPU compute
Dispatcher picks the backend by Device. CPU computes now; GPU only adds a graph node.
- 04
Autograd recording
The output TensorImpl is built, and an AddBackward node becomes its grad_fn only when needed.