Softplus

Softplus is a smooth way to turn any real-valued number into a positive output. It behaves like a softened version of ReLU: strongly negative inputs are pushed close to zero, while positive inputs pass through almost unchanged—but without ReLU’s sharp corner at zero.

How it works

The standard Softplus function is:

Softplus(x) = log(1 + exp(x))

Its derivative is the sigmoid function:

d Softplus(x) / dx = sigmoid(x)

For a large positive x, Softplus is approximately x, like ReLU.
For a large negative x, it approaches zero but never becomes exactly zero.
At zero, it has a smooth, well-defined slope rather than ReLU’s kink.

This smoothness makes Softplus useful when a network needs positive values while retaining differentiability everywhere. For example, a model can produce a positive scale, variance, rate, or standard-deviation-like parameter by applying Softplus to an unconstrained final-layer output.

Training behavior and trade-offs

Unlike a ReLU unit, a Softplus unit does not become permanently inactive: negative inputs still receive a gradient. But that gradient is sigmoid(x), which becomes extremely small for very negative inputs. A deeply negative unit can therefore learn very slowly, even though it is not literally “dead.” Softplus also requires logarithm and exponential operations, making it more expensive than ReLU’s simple thresholding.

Frameworks handle numerical stability carefully. For instance, PyTorch’s torch.nn.Softplus uses a configurable beta and switches to a linear form beyond a threshold, avoiding overflow from exp(x). Softplus is less common as the main hidden-layer activation in large networks, where ReLU-family activations are cheaper and train well, but it is a strong choice for smooth positive output constraints. Its appeal is not raw speed; it is the combination of positivity, smooth gradients, and stable optimization around zero.

Softplus is a smooth activation function defined as softplus(x) = log(1 + e^x). It approximates ReLU: it is near zero for strongly negative inputs and grows nearly linearly for positive inputs, but remains differentiable everywhere with strictly positive output. Its nonzero gradient avoids dead neurons and supports smooth optimization, though it is more computationally expensive and less sparsity-inducing than ReLU.

Think of a dimmer switch instead of a light switch. A regular ReLU activation makes a sharp choice: negative signals become zero, while positive signals pass through. Softplus does a similar job, but more gently. Rather than snapping from “off” to “on,” it smoothly brightens as the input becomes more positive.

This smoothness can make a neural network’s learning process less abrupt. It still favors useful positive signals and keeps outputs above zero, but it does not completely shut off negative ones. That can help avoid neurons becoming permanently inactive, though Softplus is usually a little slower to use than the simpler ReLU.