Tensor

Constructor is one of the six sanctioned host-to-engine bridge boundaries (rule H4). NumPy arrays cross the bridge exactly once here; afterwards the data is owned by Lucid's engine. The resulting layout is C-contiguous row-major:

where $e$ is the element size in bytes and $s_j$ are the dimension sizes.

This is an internal accessor exposed for advanced use cases such as writing custom ops that bind directly against the engine. Public APIs should compose existing Tensor operations instead. Logically the identity projection $\pi: \text{Tensor} \to \text{TensorImpl}$ with $\pi(t) = t.\_impl$.

For a tensor with $N$ elements arranged in $d$ dimensions $(s_0, s_1, \ldots, s_{d-1})$, the total element count satisfies:

The memory footprint of a tensor is $N \times e$ bytes, where $N$ is the total number of elements and $e$ is dtype.itemsize.

On Apple Silicon the CPU and GPU share the same physical DRAM (unified memory architecture). Moving a tensor between devices copies the logical dispatch target, not physical bytes, unless the tensor is in a non-shared Metal buffer. The dispatch routing rule is

For a shape tuple $(s_0, \ldots, s_{d-1})$ the rank is simply

For a tensor of shape $(s_0, s_1, \ldots, s_{d-1})$, the transposed tensor has shape $(s_{d-1}, \ldots, s_1, s_0)$. This differs from mT, which only transposes the final two axes.

For a tensor of shape $(\ldots, m, n)$ the result has shape $(\ldots, n, m)$. The leading batch dimensions are unchanged.

Equivalent predicate $\text{is\_metal}(t) = (t.\text{device} = \text{GPU})$. On Apple Silicon the Metal backend dispatches via MLX, which lazily builds a graph and evaluates on the GPU when results are observed.

On Apple Silicon, shared storage permits zero-copy hand-off between CPU and GPU code paths because both engines see the same physical DRAM page. The trade-off is that Metal's GPU caches cannot be used as aggressively as with private storage; for compute-bound GPU kernels, private mode is often preferable. Predicate: $\text{is\_shared}(t) = \mathbf{1}\{\text{storage\_mode}(t) = \text{Shared}\}$.

In the computation graph, leaf nodes are the "inputs" to the forward pass. All tensors created with requires_grad=False are leaves by definition. Tensors created with requires_grad=True directly by the user are also leaves. Tensors produced by differentiable operations on requires_grad=True inputs are not leaves — they are intermediate nodes and their .grad is not retained unless retain_grad is called. Predicate:

Tensors with requires_grad=True become nodes in the autograd DAG. Operations consuming them produce non-leaf tensors that also require gradients (transitive closure of the flag along the forward graph).

Equivalent to len(tensor.shape); matches the conventional rank $d$ such that $\text{shape} \in \mathbb{N}^d$.

The full shape and the element count are related by

A tensor of shape $(s_0, \ldots, s_{d-1})$ is contiguous iff its strides satisfy the C-order recurrence

(in element units). Many Accelerate/MLX kernels require contiguous inputs; if not, call contiguous first.

The gradient has the same shape as the tensor itself:

The graph forms the chain-rule factorisation used by backward:

where $\mathbf{y} = f(\mathbf{x})$ and grad_fn carries the Jacobian-vector product $v \mapsto v \cdot J_f$.

In-place flag flip: the underlying storage is preserved, but the TensorImpl is replaced with one whose autograd flag is $\text{requires\_grad} \in \{\text{True}, \text{False}\}$. Only valid on leaf tensors; non-leaf tensors inherit the flag from their producing op and cannot be flipped on directly.

This method must be called before the forward computation whose gradient you want to inspect. Calling it after backward has no effect. Conceptually retains $\frac{\partial \mathcal{L}}{\partial \mathbf{y}}$ for the intermediate node $\mathbf{y}$ instead of discarding it after its parents have consumed it.

• For leaf tensors the hook fires after backward accumulates the gradient, which is the common use case (gradient clipping, logging).
• For non-leaf tensors, call retain_grad before the forward pass so the gradient is preserved and available when hooks fire.
• The hook must be registered before the forward computation for non-leaf tensors; for leaf tensors any timing works.

Chain-rule effect: if the hook returns a tensor $\tilde g$, the engine substitutes $\frac{\partial \mathcal{L}}{\partial \mathbf{x}} \leftarrow \tilde g$ before continuing backward propagation. A returned None leaves the gradient untouched.

The chain rule and reverse-mode AD

Given a scalar loss $\mathcal{L}$ and a sequence of operations $\mathbf{z} = f_n(\cdots f_2(f_1(\mathbf{x})) \cdots)$, the chain rule gives:

where $J_{f_i}$ is the Jacobian of the $i$-th operation. Reverse-mode AD (backpropagation) evaluates this product right-to-left, starting from the scalar output with seed gradient $\frac{\partial \mathcal{L}}{\partial \mathcal{L}} = 1$, so it computes gradients for all inputs in a single backward pass — $\mathcal{O}(1)$ passes regardless of the number of parameters.

Gradient accumulation

Gradients are added to tensor.grad rather than overwritten. This is intentional: it supports patterns like accumulated gradient steps. Zero gradients explicitly before each optimisation step:

.. code-block:: python

for param in model.parameters():
    param.grad = None   # or param.grad.zero_()

Metal flush

On Metal devices Lucid calls TensorImpl::eval() before the backward pass. This forces MLX to evaluate the forward graph eagerly so that the backward kernel sees concrete values rather than deferred MLX computations. In practice this yields roughly 2× faster backward passes for typical model sizes.

Pointwise rectification:

The non-smooth point at $x = \text{min}$ has subgradient $[0, 1]$; autograd selects 1 for $x > \text{min}$ and 0 otherwise.

Pointwise saturation:

The gradient is the indicator $\mathbf{1}\{x_i < \text{max}\}$.

Triggers a device-to-host synchronisation when called on a Metal tensor — the value cannot be inspected until any pending MLX computation has completed. Avoid calling in tight loops on GPU tensors; prefer tensor.cpu().tolist() for batch extraction.

Defined only when $\text{numel} = \prod_i s_i = 1$. item is one of the sanctioned engine-to-host bridge points (rule H4) and detaches from autograd: the returned Python scalar carries no gradient information.

This is one of the sanctioned bridge points between Lucid and the outside world (see project rule H4). The returned ndarray does not participate in autograd; downstream NumPy operations will not produce gradients.

Layout is C-contiguous; for a tensor of shape $(s_0, \ldots, s_{d-1})$ and itemsize $e$, the NumPy strides equal

Numpy-free for every supported dtype (F16 / F32 / F64 / I8 / I16 / I32 / I64 / Bool / C64). Delegates to the engine-side TensorImpl.tolist(), which mirrors item()'s dtype dispatch but walks the full shape recursively and yields Python complex for C64 leaves. Forces a device-to-host synchronisation for Metal tensors via the underlying to_bytes() snapshot. Autograd information is dropped — the returned Python objects are pure value copies.

Unfold is the fundamental primitive behind 1-D convolution and sliding-window aggregations. The windows may overlap when step < size.

Aliases the same storage as self with the gradient flag cleared. Mathematically the same identity as detach — equal values, zero Jacobian — but unlike detach writes through data propagate back to self's underlying buffer without participating in autograd. Prefer detach for new code; data is retained for API compatibility.

Allocates $\prod_i s_i \cdot e$ bytes of uninitialised memory where $e$ is dtype.itemsize. Faster than new_zeros because the engine skips the zero-fill kernel. Must be followed by a write before any read.

Initialises every element to the additive identity $0$. The zero-fill is delegated to a fused engine kernel — Accelerate catlas_*set on CPU and MLX broadcast-fill on Metal.

Initialises every element to the multiplicative identity $1$. For arbitrary fill values use new_full.

Constant tensor $f \cdot \mathbf{1}$ where $f$ is fill_value and $\mathbf{1}$ is the all-ones tensor of the given shape. fill_value is promoted to dtype before the broadcast fill.

Always copies — never aliases data's storage even when data is already a Tensor. Routes through the same _to_impl bridge boundary as the public constructor (rule H4).

The total memory footprint of the tensor is:

Total footprint of the tensor satisfies $\text{nbytes} = \text{numel} \cdot \text{itemsize}$.

where $s_i$ are the dimension sizes and $e$ is the element size in bytes.

For a C-contiguous tensor of shape $(s_0, \ldots, s_{d-1})$ the element strides satisfy the row-major recurrence

Non-contiguous tensors (e.g. transposed or sliced views) may have arbitrary strides; the address of element [i_0, \ldots, i_{d-1}] is base + \sum_k i_k \cdot \text{stride}[k].

On Apple Silicon CPU and GPU share unified DRAM, so the same $\text{data\_ptr}(t)$ identifies a buffer addressable from both backends. The value is stable for the lifetime of self; equality is the canonical aliasing predicate $t_1 \sim t_2 \iff \text{data\_ptr}(t_1) = \text{data\_ptr}(t_2)$.

In Lucid every tensor owns a fresh contiguous storage, so the offset is identically zero: $\text{offset}(t) \equiv 0$. Frameworks that support sub-views over a shared buffer use this for pointer arithmetic; for Lucid it is provided purely for API compatibility.

Equivalent to x.conj().mT for complex tensors. For batched linear algebra the leading dimensions are untouched.

Provided purely for API compatibility with code that uses legacy type strings. New code should use dtype for inspection and to for casting — both avoid stringly-typed dispatch.

Indicator-style encoding: $\text{idx}(t) = 0 \cdot \mathbf{1}\{\text{is\_metal}\} + (-1) \cdot \mathbf{1}\{\text{is\_cpu}\}$. Lucid only supports a single Metal device on Apple Silicon, so a positive index is always 0.

See also is_pinned (always False) and share_memory_ (also a no-op). The function is the identity: $\text{pin\_memory}(t) \equiv t$.

Identically false: $\text{is\_pinned}(t) \equiv \text{False}$. Unified memory makes the concept moot — all host buffers are already DMA-accessible to the GPU without page-locking.

Identically false: $\text{is\_cuda}(t) \equiv \text{False}$. Use is_metal to query GPU residency on Apple Silicon.

Reshape preserves element order under row-major (C) traversal; the element at flat index $k = \sum_i i_k \cdot \prod_{j>k} s_j$ in self becomes the element at the same flat index in the result. A view is returned when the source is contiguous; otherwise a contiguous copy is materialised first.

Lucid does not currently expose a fully-featured Storage type; untyped_storage is the introspection-only minimum. Mutating through this handle is not supported. The reported size satisfies $|\text{storage}| = \text{numel} \cdot \text{itemsize}$.

Provided for API compatibility. See also is_pinned and pin_memory, which are no-ops for the same reason. The operation is the identity: $\text{share\_memory\_}(t) \equiv t$.

In-place operations bypass autograd's view tracking for performance. Calling fill_ on a leaf tensor with requires_grad=True may raise a runtime error from the autograd engine.

Mathematically the result is the constant tensor $v \cdot \mathbf{1}$ with the same shape as self; every entry satisfies $x_i \leftarrow v$.

Unlike clone, copy_ does not allocate a new tensor — only self's storage is written. The autograd graph is not extended by this operation. Element-wise:

with standard right-aligned broadcasting from other.shape to self.shape.

Broadcasting rules follow the standard right-aligned semantics: each dimension of self.shape must either equal the corresponding entry of other.shape or be 1. Size-1 axes are stretched by setting the corresponding stride to zero, so the resulting view aliases the source storage. Formally, for each axis

with stride $0$ on stretched axes.

For a contiguous tensor the new strides are computed as

so the layout is row-major with no data motion.

Values are cast element-wise: $x_i \leftarrow \text{cast}_{\tau}(x_i)$ where $\tau = \text{other.dtype}$. Casts between floating-point types preserve gradients; integer→float→integer round-trips lose information in the integer truncation step.

As with fill_, in-place mutation bypasses autograd's view tracking. Calling zero_ on a leaf tensor with requires_grad=True may raise a runtime error from the autograd engine.

Element-wise: $x_i \leftarrow 0$ for every position. The result is the additive identity of the tensor algebra at the same shape and dtype.

No-op when the source already matches the target device and dtype, unless copy=True; in that case the same Tensor instance is returned without allocation. Apple Silicon's unified memory means device transfers between CPU and Metal copy the dispatch label but share physical DRAM when the underlying buffer is a SharedStorage.

No-op when the tensor is already on Metal. Under Apple Silicon's unified memory architecture the physical DRAM is shared with the CPU; this call only updates the dispatch target so subsequent ops run through the MLX/Metal backend.

No-op when the tensor is already on CPU. On Apple Silicon both CPU and Metal share the same physical DRAM (unified memory); this call relabels the dispatch target so subsequent ops route through Apple Accelerate (vDSP / vForce / BLAS / LAPACK).

Returns self unchanged when the source dtype is already float32. Otherwise performs a copy-cast via the engine's astype op; the device is preserved.

Returns self unchanged when the source dtype is already float64. Otherwise allocates a new tensor with widened precision; device is preserved.

Returns self unchanged when the source dtype is already float16. Lossy narrowing from float32/float64 may overflow to inf for magnitudes above ~65504 and underflow to 0 for magnitudes below ~6e-5.

Returns self unchanged when the source dtype is already int32. Casts from floating point truncate toward zero; values outside [-2**31, 2**31 - 1] produce undefined results.

Returns self unchanged when the source dtype is already int64. Casts from floating point truncate toward zero. Typically used for index tensors (e.g. gather / scatter).

Returns self unchanged when the source dtype is already bool. Otherwise each element is reduced to True if nonzero and False if zero.

Mirrors NumPy/sequence semantics: $\text{len}(t) = s_0$. Iterating with __iter__ yields exactly $s_0$ slices.

This matches the convention adopted by every mainstream tensor framework. The error guards against subtle bugs such as if pred_tensor: silently truncating to the first element.