class

MishBackward

MishBackward

source edit

Python wrappers

Autograd node for Mish $y = x \cdot \tanh(\mathrm{softplus}(x))$ (Misra 2019).

The composed function's derivative involves both $\tanh$ and its derivative through the softplus, so the backward is delegated to the backend where it runs as a single fused kernel.

Math

$y = x\,\tanh(\mathrm{softplus}(x)),$

= \tanh(\mathrm{softplus}(x)) + x\,\sigma(x)\,(1 - \tanh^2(\mathrm{softplus}(x)))$$ **References** Misra, "Mish: A Self Regularized Non-Monotonic Activation Function", [arXiv:1908.08681](https://arxiv.org/abs/1908.08681).

Attributes

schema_v1OpSchema

"mish", AmpPolicy::ForceFP32.

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::mish to compute $y = x \cdot \tanh(\mathrm{softplus}(x))$ .

Methods

grad_formula

int grad_formula(const int & g)

source edit

Eager-mode backward: dispatches to IBackend::mish_backward.

class

MishBackward

MishBackward

source edit

Python wrappers

Autograd node for Mish $y = x \cdot \tanh(\mathrm{softplus}(x))$ (Misra 2019).

The composed function's derivative involves both $\tanh$ and its derivative through the softplus, so the backward is delegated to the backend where it runs as a single fused kernel.

Math

$y = x\,\tanh(\mathrm{softplus}(x)),$

= \tanh(\mathrm{softplus}(x)) + x\,\sigma(x)\,(1 - \tanh^2(\mathrm{softplus}(x)))$$ **References** Misra, "Mish: A Self Regularized Non-Monotonic Activation Function", [arXiv:1908.08681](https://arxiv.org/abs/1908.08681).

Attributes

schema_v1OpSchema

"mish", AmpPolicy::ForceFP32.

Static methods

static

dispatch

int dispatch(backend::IBackend & be, const int & a, const int & s, Dtype dt)

source edit

Forward — calls IBackend::mish to compute $y = x \cdot \tanh(\mathrm{softplus}(x))$ .

Methods

grad_formula

int grad_formula(const int & g)

source edit

Eager-mode backward: dispatches to IBackend::mish_backward.