Softmax

Softmax turns a model’s raw class scores into a set of probabilities that compete with one another. Instead of asking “how likely is each class independently?”, it asks “how should 100% of belief be divided among these choices?”

From scores to probabilities

A network usually produces unrestricted numbers called logits in its final layer. For logits \(z_1,\ldots,z_K\), softmax assigns class i the value \(e^{z_i}/\sum_j e^{z_j}\). Exponentiation makes larger scores count disproportionately more, while division by the total ensures every output is between 0 and 1 and all outputs sum to 1. Raising one logit also lowers the other probabilities: softmax models a mutually exclusive choice.

Logits [2, 1, 0] become probabilities roughly [0.67, 0.24, 0.09].
Adding the same number to every logit changes nothing, because only their relative differences matter.

Why it trains well with cross-entropy

Softmax is commonly paired with cross-entropy loss. For a one-hot target, the gradient with respect to each logit simplifies to predicted probability minus target. A class assigned 0.80 when it should be 0 loses gradient pressure; a true class assigned 0.01 receives a strong corrective signal. This clean gradient is why classifiers are normally trained with a combined “softmax cross-entropy” operation rather than applying softmax and a loss separately.

Practical details and failure modes

Never compute exponentials naïvely for huge logits: \(e^{1000}\) overflows. Frameworks such as PyTorch’s CrossEntropyLoss take logits directly and use the stable trick of subtracting the largest logit first. Softmax is mainly an output operation, not a hidden-layer activation: its outputs saturate near 0 or 1 and tightly couple units. For extremely large class sets, calculating every probability costs proportional to the number of classes, motivating sampled or approximate alternatives.

Softmax is a function that converts a vector of real-valued scores (logits) into positive values that sum to 1, producing a categorical probability distribution. Each output reflects a class’s relative score: increasing one logit raises its probability while lowering the others. It is used at multi-class output layers and in attention weighting. Softmax enables probability-based losses and interpretable normalized predictions; numerically stable implementations prevent overflow during training.

Imagine a quiz show with several possible answers. Before the AI commits, it may have a different “confidence score” for each choice. Softmax turns those rough scores into a neat set of probabilities: numbers that add up to 100%.

For example, an image-recognition system might decide a photo is 80% likely to show a cat, 15% likely to show a fox, and 5% likely to show a dog. Softmax makes the choices easy to compare and gives the system a clear way to express uncertainty. It is often used at the final step when an AI must choose one label from a fixed list, such as identifying a language, emotion, or object.