Sigmoid

The sigmoid is a smooth “squashing” function: it takes any real-valued number and converts it to a value strictly between 0 and 1. That makes it feel naturally suited to questions with two outcomes, such as “is this feature present?” or “should this gate let information through?”

What it computes
For an input x, sigmoid computes σ(x) = 1 / (1 + e^−x). Large positive inputs become values near 1; large negative inputs become values near 0; and zero maps to 0.5. A neuron first forms a weighted sum, then sigmoid turns that score into a bounded activation. In a binary classifier, the final sigmoid output can be interpreted as a probability when the model is trained appropriately. The raw score before sigmoid is called a logit.

Why saturation causes trouble
Sigmoid’s useful learning region is near zero. Its derivative is σ(x)(1 − σ(x)), with a maximum of only 0.25. When the input is strongly positive or negative, the output flattens near 1 or 0 and the derivative approaches zero. During backpropagation, those tiny gradients are multiplied across layers, so an early layer in a deep network can receive almost no learning signal.

For hidden layers, modern networks usually prefer ReLU, GELU, or related activations because they preserve stronger gradients over a wider range.
For the final layer of binary classification, sigmoid remains the standard choice because its bounded output has the right meaning.
In an LSTM, sigmoid is valuable in gates: values near 0 block information and values near 1 preserve it.

Practical training detail
Do not manually apply sigmoid before a binary cross-entropy loss when your framework provides a logits-based version. PyTorch’s BCEWithLogitsLoss, for example, combines sigmoid and cross-entropy using a numerically stable calculation. Applying a separate sigmoid can turn extreme logits into rounded 0s or 1s, creating unstable losses or weak gradients. Sigmoid is inexpensive to evaluate, but its saturation—not its compute cost—is why it is rarely the default hidden-layer activation in deep networks.

Sigmoid is a smooth, saturating activation function, σ(x) = 1/(1 + e^−x), that maps any real-valued input to the range (0, 1). It is useful for representing independent probabilities or gates. Its gradients become near zero for large positive or negative inputs, causing vanishing gradients and making deep networks difficult to train when used in hidden layers.

Think of a sigmoid as a smooth dimmer switch. No matter how strong or weak an incoming signal is, the switch turns it into a value between 0 and 1. Very negative signals become close to 0, very positive ones become close to 1, and signals near the middle land somewhere in between.

This makes sigmoid useful when a network needs an answer that feels like a probability: “How likely is this email spam?” or “Is this image likely to contain a cat?” A result near 1 means “very likely”; near 0 means “unlikely.” Its gentle curve also lets the network express uncertainty rather than only yes-or-no answers.