Notes
🔍

Target Encoding

When your data has categories like “city,” “product type,” or “browser,” your model still needs numbers. Target encoding is a way to turn each category into a number that reflects what that category tends to predict.

What target encoding does

In target encoding, you replace a categorical value with a statistic of the target computed from training data. For regression, that statistic is usually the mean target; for binary classification, it’s often the class probability (mean of 0/1). Example: in credit scoring, if applicants from “Region A” default 8% of the time in the training set, “Region A” might be encoded as 0.08.

Why it’s powerful (and risky)

Unlike one-hot encoding, target encoding can handle high-cardinality features (thousands of categories) without exploding the number of columns, and it gives the model a strong, direct signal. The risk is target leakage: if you compute the encoding using the same rows you train on, the model can “peek” at the answer, especially for rare categories. The fix is to compute encodings in an out-of-fold way and to use smoothing toward the global mean:

  • Out-of-fold encoding: for each fold, compute category statistics on the other folds, then encode the held-out fold.
  • Smoothing / regularization: blend a category’s mean with the overall mean, weighting by category count.
  • Noise (optional): add small random jitter to reduce overfitting.

Where you’ll see it in practice

It’s common in churn prediction, fraud detection, and demand forecasting with features like merchant ID, ZIP code, or product SKU. In Python, you’ll encounter it via libraries like category_encoders (e.g., TargetEncoder) or as “mean encoding” in Kaggle-style pipelines.

Target encoding is a categorical feature encoding method that replaces each category with a statistic of the supervised target computed from the training data (for example, the mean label for regression or positive-class rate for classification), typically with smoothing and out-of-fold computation. It matters because it converts high-cardinality categories into informative numeric features while controlling target leakage; done incorrectly, it inflates validation scores and harms generalization.

Imagine you run a café and want to predict tomorrow’s sales. You know the neighborhood matters, but “Downtown” and “Riverside” aren’t numbers a model can easily use. Target Encoding solves this by replacing each category with a helpful summary of the thing you’re trying to predict (the “target”).

For example, if emails from “Company A” are spam 80% of the time, that sender might be encoded as 0.8. If houses in one zip code usually sell for more, that zip code gets a higher value. It’s a simple way to turn labels into numbers that carry real predictive meaning.