LstmBackward

Autograd node for the single-layer LSTM with BPTT backward.

Implements the standard Hochreiter & Schmidhuber (1997) LSTM gating equations:

where $\sigma$ is the logistic sigmoid, $\odot$ is the element-wise (Hadamard) product, and the four gates $i, f, g, o$ are stored in that exact concatenation order along the leading axis of gates_all (i.e. gates_all[..., 0:H] is the input-gate pre-activation, gates_all[..., H:2H] the forget gate, and so on).

Inherits directly from Node rather than FuncOp because the 7-edge topology {input, h0, c0, wih, whh, bih, bhh} is built by hand: each tensor that requires a gradient gets an explicit AccumulateGrad leaf wired in forward(), and non-differentiable tensors (such as biases when has_bias=false) get a null edge. Only the output sequence res[0] carries this node as its grad_fn; hn and cn are detached so they can be reused as initial states for a subsequent stacked layer in pure Python without double-counting gradients.

Math

The BPTT backward unrolls $T$ time steps in reverse, accumulating the gradient contributions through both the cell-state path $\partial c_t / \partial c_{t-1} = f_t$ and the gate path $\partial c_t / \partial \{i_t, f_t, g_t\}$, then maps cell-state gradients back through the sigmoid / tanh derivatives to give per-time-step weight gradients $\partial \mathcal{L}/\partial W_{ih}$, $\partial \mathcal{L}/\partial W_{hh}$, $\partial \mathcal{L}/\partial b_{ih}$, $\partial \mathcal{L}/\partial b_{hh}$. The full unrolled BLAS path lives in IBackend::lstm_backward.

References

Hochreiter & Schmidhuber, "Long Short-Term Memory" (Neural Computation 1997). Hochreiter, "Untersuchungen zu dynamischen neuronalen Netzen" (1991) for the original analysis of the vanishing-gradient problem that motivates the gating mechanism.