Notes
🔍

Cross-Entropy Loss

When a classifier says “this email is spam with 90% confidence,” you want a training signal that rewards being confidently right and strongly penalizes being confidently wrong. Cross-entropy loss is the standard way to do that when your model outputs probabilities.

What it measures (and why it feels natural)

Cross-entropy loss compares the true label distribution to the model’s predicted probability distribution. For a single example in binary classification (label y in {0,1}, predicted probability p for class 1), it is:

L = -[ y * log(p) + (1 - y) * log(1 - p) ]

If the true label is 1 and the model predicts p = 0.99, the loss is tiny; if it predicts p = 0.01, the loss is huge. That “log penalty” is what makes overconfident mistakes hurt a lot.

How it’s used in training

In multiclass classification, cross-entropy uses a softmax over class scores (logits) and penalizes the negative log-probability assigned to the correct class. Minimizing it via gradient descent pushes probability mass toward the right label and away from the others. In practice you’ll see it as:

  • Log loss in scikit-learn (e.g., `sklearn.metrics.log_loss`).
  • CrossEntropyLoss in PyTorch (expects logits; applies softmax internally).

Why it matters in real problems

Cross-entropy is central in tasks like spam detection, fraud detection, and disease diagnosis because it trains models to produce meaningful probabilities, not just correct/incorrect labels. If you used a loss that doesn’t care about probability quality, you can end up with a classifier that ranks cases correctly but gives poorly calibrated confidence—bad news when decisions depend on thresholds, costs, or risk.

Cross-Entropy Loss is a classification objective that measures the negative log-likelihood of the true label under a model’s predicted class probabilities (e.g., from a softmax or sigmoid). It penalizes confident wrong predictions sharply and rewards assigning high probability to the correct class. It matters because it provides a principled, differentiable training signal for probabilistic classifiers, enabling stable gradient-based optimization and well-calibrated probability estimates.

Think of a weather app that says there’s a 90% chance of rain, but it stays sunny. That’s worse than saying 55% and being wrong, because the first claim was made with much more confidence. Cross-Entropy Loss is a way to score how “painful” a model’s predictions are in that same sense.

In classification (like spam vs. not spam), the model often outputs probabilities for each label. Cross-Entropy Loss gives a small penalty when the correct label gets a high probability, and a big penalty when the model confidently backs the wrong label. This pushes the model to be both accurate and honest about uncertainty.