class

SiluBackward

SiluBackward

source edit

Python wrappers

pylucid.nn.functional.silufunction

Autograd node for the SiLU / Swish activation $y = x \cdot \sigma(x)$ (Hendrycks & Gimpel 2016 / Ramachandran et al. 2017).

Saves the input $x$ ; the backward delegates to the backend so the fused gradient kernel ( $\sigma(x)\,(1 + x(1 - \sigma(x)))$ ) runs as a single pass on MLX or a single Accelerate composition on CPU rather than the prior seven-op storage-primitive expansion.

Math

= \sigma(x)\,\bigl(1 + x\,(1 - \sigma(x))\bigr)$$ **References** Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)", 2016. Ramachandran et al., "Searching for Activation Functions", 2017.

Attributes

schema_v1OpSchema

"silu", AmpPolicy::Promote.

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::silu to compute $y = x \cdot \sigma(x)$ .

Methods

grad_formula

int grad_formula(const int & g)

source edit

Eager-mode backward: dispatches to IBackend::silu_backward with the saved input and the upstream gradient.

class

SiluBackward

SiluBackward

source edit

Python wrappers

pylucid.nn.functional.silufunction

Autograd node for the SiLU / Swish activation $y = x \cdot \sigma(x)$ (Hendrycks & Gimpel 2016 / Ramachandran et al. 2017).

Math

= \sigma(x)\,\bigl(1 + x\,(1 - \sigma(x))\bigr)$$ **References** Hendrycks & Gimpel, "Gaussian Error Linear Units (GELUs)", 2016. Ramachandran et al., "Searching for Activation Functions", 2017.

Attributes

schema_v1OpSchema

"silu", AmpPolicy::Promote.

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::silu to compute $y = x \cdot \sigma(x)$ .

Methods

grad_formula

int grad_formula(const int & g)

source edit

Eager-mode backward: dispatches to IBackend::silu_backward with the saved input and the upstream gradient.