functional

Apply the rectified linear unit function element-wise.

Leaky rectified linear unit activation.

Exponential linear unit activation.

Continuously-differentiable Exponential Linear Unit (Barron 2017).

Scaled exponential linear unit activation.

Gaussian Error Linear Unit activation.

Sigmoid Linear Unit (a.k.a. Swish-1) activation.

Mish activation function.

Hard Swish activation — piecewise linear approximation of `silu`.

Hard sigmoid — piecewise linear approximation of `sigmoid`.

Logistic sigmoid activation.

Hyperbolic tangent activation.

Apply the softmax function along a dimension.

Numerically stable log-softmax along a dimension.

Smooth approximation of `relu`.

Rectified linear unit clipped at six.

Softmin — softmax applied to the negation of the input.

Gated Linear Unit (Dauphin et al. 2017).

Parametric Rectified Linear Unit (He et al. 2015).

Hard shrinkage operator.

Tanh shrinkage activation.

Soft shrinkage operator — proximal operator of $\lambda \|\cdot\|_1$.

Normalize a tensor to unit $L_p$ norm along a dimension.

Cosine similarity between two tensors along a dimension.

Element-wise $L_p$ distance between two equally-shaped tensors.

Hardtanh — element-wise clamp to $[\text{min\_val}, \text{max\_val}]$.

Numerically stable $\log \sigma(x)$.

Softsign activation.

Threshold activation — gate elements by a scalar cutoff.

Randomised leaky ReLU (Xu et al. 2015).

Gumbel-Softmax — differentiable relaxation of categorical sampling.

Bilinear transformation applied to two inputs.

Fused linear + ReLU forward kernel.

Fused linear + GELU forward kernel.

1-D cross-correlation over batched 3-D input.

2-D cross-correlation over batched 4-D input.

3-D cross-correlation over batched 5-D input.

Transposed 1-D convolution (a.k.a. "fractionally-strided" conv).

Transposed 2-D convolution — the standard upsampling primitive.

Transposed 3-D convolution — volumetric upsampling.

Batch normalization (Ioffe & Szegedy, 2015).

Layer normalization (Ba, Kiros & Hinton, 2016).

Group normalization (Wu & He, 2018).

Root-mean-square layer normalization (Zhang & Sennrich, 2019).

Instance normalization (Ulyanov, Vedaldi & Lempitsky, 2016).

Local response normalization (Krizhevsky, Sutskever & Hinton, 2012).

1-D max pooling over a sliding window.

2-D max pooling over a sliding window.

3-D max pooling over a sliding window.

1-D average pooling over a sliding window.

2-D average pooling over a sliding window.

3-D average pooling over a sliding window.

1-D adaptive average pooling — produces a fixed output length.

2-D adaptive average pooling — produces a fixed `(H, W)`.

3-D adaptive average pooling — produces a fixed `(D, H, W)`.

2-D adaptive max pooling — fixed-shape `(H, W)` via per-cell max.

1-D adaptive max pooling — produces a fixed output length.

3-D adaptive max pooling — produces a fixed `(D, H, W)`.

1-D Lp-norm pooling — $\big(\sum |x|^p\big)^{1/p}$.

2-D Lp-norm pooling — $\big(\sum |x|^p\big)^{1/p}$.

3-D Lp-norm pooling — $\big(\sum |x|^p\big)^{1/p}$.

Inverse of `max_pool1d` via scatter at saved argmax indices.

Inverse of `max_pool2d` via scatter at saved argmax indices.

Inverse of `max_pool3d` via scatter at saved argmax indices.

Fractional max-pooling over a 2-D input (Graham, 2014).

Fractional max-pooling over a 3-D input (Graham, 2014).

Channel-wise dropout for 1-D sequence / feature inputs.

Channel-wise dropout for 2-D (spatial) feature maps.

Channel-wise dropout for 3-D (volumetric) feature maps.

Alpha dropout — variance-preserving dropout for SELU networks.

Channel-wise alpha dropout for SELU convolutional networks.

Scaled dot-product attention — the core of every Transformer block.

Mean-squared-error (L2) loss between input and target.

Mean-absolute-error (L1) loss between input and target.

Smooth L1 loss — a quadratic-near-zero, linear-far-from-zero hybrid.

Huber loss — robust regression with a tunable transition point.

Cross-entropy loss for multi-class classification.

Negative log-likelihood loss for multi-class classification.

Binary cross-entropy between predicted probabilities and targets.

Binary cross-entropy from raw logits (numerically stable).

Kullback-Leibler divergence between two distributions.

Triplet margin loss for metric learning.

Triplet margin loss with a user-supplied distance function.

Cosine embedding loss for pairwise similarity learning.

Pairwise ranking hinge loss.

Hinge embedding loss.

Poisson negative log-likelihood loss for count regression.

Gaussian negative log-likelihood for heteroscedastic regression.

Connectionist Temporal Classification (CTC) loss.

Multi-class hinge (margin) loss — Crammer-Singer SVM objective.

Multi-label hinge loss for set-valued targets.

Logistic (softplus) loss for binary classification with ±1 labels.

Per-class logistic loss averaged over labels (multi-label BCE).

Look up rows of an embedding table by integer indices.

One-hot encode an integer class index tensor.

Apply Rotary Position Embedding (RoPE) to query and key tensors.

Build the 1-D sinusoidal positional encoding table from "Attention Is All You Need".

Build the 2-D sinusoidal positional encoding from DETR (Carion et al., 2020).

Resample an N-D tensor to a target spatial size or scale factor.

Sample an input feature map at flow-field coordinates.

Generate a sampling grid from a batch of affine transform matrices.

Pad an N-D tensor along an arbitrary set of trailing dimensions.

Extract sliding local blocks (im2col) from a batched 4-D tensor.

Combine an array of sliding local blocks back into an image (col2im).

Aggregate embeddings into per-bag pooled vectors.

Sub-pixel upsampling: rearrange channels into spatial resolution.

Inverse of `pixel_shuffle`: pack spatial blocks into channels.

Stateless functional multi-head attention forward pass.

Group-then-transpose channel rearrangement (ShuffleNet).

Pairwise $L_p$ distances between rows of a 2-D tensor.