Notes
🔍

Log-Cosh Loss

When you train a regression model, you’re really teaching it what “being wrong” should feel like. Log-cosh loss is a way to measure prediction error that behaves gently for small mistakes but refuses to be bullied by extreme outliers.

What it is (and the shape intuition)

Log-cosh loss applies the function log(cosh(·)) to the residual r = y - ŷ. The key idea is its “two personalities”:

  • For small residuals, it looks like mean squared error (MSE): it’s roughly proportional to r², which gives smooth gradients and encourages precise fits.
  • For large residuals, it looks like mean absolute error (MAE): it grows roughly like |r|, so a few huge errors don’t dominate training the way MSE does.

This makes it a robust loss: it balances sensitivity and stability without a hard “kink” in the gradient.

Why it matters in supervised regression

Loss choice directly shapes what the optimizer prioritizes. With MSE, a single bizarre data point (bad sensor reading, data entry error) can pull the model strongly. With MAE, training can be less smooth because the gradient changes abruptly at zero. Log-cosh keeps optimization smooth like MSE while reducing outlier influence like MAE—often leading to steadier training and more reliable predictions.

Where you’ll see it in practice

  • House price prediction where a few luxury homes are extreme outliers.
  • Demand forecasting
  • Credit loss estimation

In TensorFlow/Keras you’ll encounter it as tf.keras.losses.LogCosh(), a drop-in alternative to MSE/MAE for regression.

Log-Cosh Loss is a regression objective defined as the mean of log(cosh(y − ŷ)), where y is the target and ŷ the prediction. It behaves like squared error for small residuals and like absolute error for large residuals, yielding smooth gradients while reducing sensitivity to outliers. It matters because it can improve robustness and optimization stability compared with pure MSE or MAE.

Imagine you’re grading dart throws at a target. If a dart lands a little off, you subtract a few points. But if one dart flies wildly off, you don’t want that single mistake to dominate the whole scorecard. Log-Cosh Loss is a way to “score” prediction errors in regression (predicting numbers like house prices or delivery times) with that kind of balance.

It treats small mistakes much like the usual “squared error” scoring, encouraging accurate predictions. But for big mistakes (often caused by odd, rare data points), it grows more gently, so outliers don’t overwhelm training.