Swish

Swish is an activation function that gives a neuron a smooth way to decide how much of its input should pass onward. Like ReLU, it preserves strong positive signals, but it avoids ReLU’s hard corner at zero and does not completely shut off negative inputs.

How it works
Swish is defined as f(x) = x · sigmoid(βx), where the sigmoid term lies between 0 and 1 and acts like a soft gate. In the common fixed form, β = 1, Swish is also called SiLU (Sigmoid Linear Unit). Large positive values pass through almost unchanged because sigmoid(βx) approaches 1. Large negative values are suppressed because the gate approaches 0. Around zero, the transition is smooth rather than abrupt.

Why its shape helps training
Unlike ReLU, Swish is smooth and has a small region where increasing an already-negative input produces a slightly lower output. This non-monotonic “dip” sounds odd, but empirical studies found it can help deep networks optimize. Its derivative changes gradually, giving backpropagation a less jagged gradient signal near zero.

• ReLU outputs exactly zero for negative inputs, so a unit can become permanently inactive if its gradient remains zero.
• Swish retains a small gradient for moderately negative inputs, so such units can still adjust.
• For very negative inputs, the sigmoid gate saturates and gradients still fade; Swish reduces, rather than eliminates, this problem.

Use in real networks
Swish became prominent in EfficientNet and appears in many modern convolutional and transformer-style blocks. In PyTorch, the usual construct is torch.nn.SiLU(). It costs more than ReLU because it evaluates a sigmoid and does not produce ReLU’s exact zero outputs, which can reduce computational sparsity. In exchange, training can be smoother and reach a better solution. A loss curve that stalls with poorly behaving ReLUs near a sensitive layer can improve after switching to SiLU, though learning rate, normalization, and initialization still determine whether the run remains stable.