Mish

Mish is a smooth activation function: a small mathematical gate placed after a layer’s weighted sum. Like ReLU, it lets strong positive signals pass through, but it handles negative values and the transition around zero more gently, which can make gradient-based training behave more smoothly.

How it works
Mish is defined as x · tanh(softplus(x)), where softplus(x) = log(1 + e^x). For a large positive input, softplus is large and tanh approaches 1, so Mish becomes close to x. For a negative input, Mish does not clamp it to exactly zero as ReLU does; it produces a small negative output instead. Its curve is smooth everywhere, including at zero, so its derivative changes continuously.

Why that shape helps
A ReLU neuron has zero gradient for negative inputs. If it remains negative for all examples, its weights stop receiving useful updates: the familiar “dead ReLU” problem. Mish preserves a gradient in that region. Its slight non-monotonic dip for modestly negative inputs also changes the distribution of signals passed to the next layer. These properties have produced empirical gains in some deep convolutional networks, including early versions of YOLOv4, though Mish is not a universal replacement for ReLU or GELU.

Training trade-offs

Benefit: smooth gradients can support stable optimization in deep networks.
Cost: softplus and tanh are substantially more expensive than ReLU’s simple maximum operation, and can increase training and inference time.
Practical use: PyTorch provides torch.nn.Mish. It is a deliberate architectural choice; test it against ReLU, SiLU/Swish, or GELU under the same learning-rate schedule rather than assuming a smoother curve guarantees lower loss.

Mish is a smooth, non-monotonic activation function defined as x · tanh(softplus(x)), where softplus(x) = log(1 + e^x). Like ReLU, it preserves positive signals, but it also provides nonzero, smooth gradients for negative inputs. This can improve gradient flow and optimization stability in deep networks, though Mish costs more to compute than simpler activations such as ReLU.

Imagine a dimmer switch rather than a simple on/off light switch. A network needs ways to decide how strongly each small clue should matter: not just “use this” or “ignore this,” but everything in between.

Mish is one of those decision rules. It gently reshapes the signals moving through a neural network, keeping useful information flowing while softening less helpful signals. Its smooth behavior can make learning feel less abrupt than older on/off-style rules.

In practice, Mish is one option among several. Some deep networks learn a little better with it, especially on certain tasks, though it can require more computation than simpler choices. It helps give a model the subtlety to recognize patterns that are rarely just black and white.