Hard Sigmoid

A hard sigmoid is a deliberately simplified version of the familiar sigmoid activation. Instead of a smooth S-shaped curve built from an exponential, it uses straight lines and flat plateaus: cheaper to compute, easy to reason about, but harsher on gradients.

What it computes
A common form is hard sigmoid:

h(x) = clip(0.2x + 0.5, 0, 1)

This means it rises linearly through a central region, then clamps to 0 for sufficiently negative inputs and 1 for sufficiently positive ones. Different libraries use equivalent-looking slopes and thresholds; for example, PyTorch’s Hardsigmoid uses clip(x / 6 + 0.5, 0, 1). Like a regular sigmoid, its output stays between 0 and 1, making it useful when a network needs a gate-like value: “block,” “pass,” or something in between.

Gradient behavior and training
Inside the linear region, the derivative is a constant (0.2 in the first formula), so gradients pass backward predictably. Outside it, the derivative is exactly zero. A unit pushed far into either plateau cannot adjust through gradient descent until another part of the network moves its input back into range. This is a sharper version of the saturation problem in ordinary sigmoid, whose gradients become tiny rather than exactly zero.

Where it is useful

Gated recurrent layers and lightweight architectures can use hard sigmoid for inexpensive gate calculations.
Mobile or quantized models benefit because clipping and simple arithmetic are easier than exponentials on constrained hardware.
It is less suitable for deep hidden stacks: many saturated units can create dead regions and stall learning.

For a gate, this trade-off is sensible: a nearly binary decision is desirable. For a general hidden activation, smoother or non-saturating choices such as ReLU-family activations usually preserve learning signals more reliably.

Hard sigmoid is a piecewise-linear approximation to the logistic sigmoid activation: it maps inputs to a bounded range, typically 0 to 1, with a linear central region and flat saturation regions at both ends. It is cheaper to compute than standard sigmoid, but its zero gradients outside the central interval can halt learning for saturated units. It is useful where bounded gating signals are needed with low computational cost.

Imagine a dimmer switch for a light: turn it far left and the light is fully off; turn it far right and it is fully on; in between, it brightens steadily. A Hard Sigmoid does something similar inside a neural network.

It takes a value and squeezes it into a simple range, usually from 0 to 1. Very low values become 0, very high ones become 1, and middle values are mapped along a straight slope. This gives the network a clear “off,” “on,” or “partly on” signal.

It is a simpler, faster-to-calculate version of the smoother sigmoid function, useful when speed matters more than perfectly smooth transitions.