Data Leakage

It’s surprisingly easy to build a model that looks brilliant in your notebook but fails in the real world. A common reason is that the model accidentally got access to information it wouldn’t have at prediction time.

What “data leakage” really means

Data leakage happens when information from outside the training data—especially from the validation/test set or from the future—sneaks into the training process. That “extra” information makes evaluation metrics look better than they should, because the model is being graded on a test it has effectively already seen. Leakage can be obvious (using the target directly) or subtle (using a feature that encodes the target indirectly).

Common ways leakage shows up

• Preprocessing before splitting: fitting a scaler, imputer, or PCA on the full dataset, then splitting. The transformation parameters contain test-set information.
• Target leakage features: variables created after the outcome occurs (e.g., “refund issued” when predicting fraud, or “cancellation date” when predicting churn).
• Time leakage: random splits on time-ordered data (e.g., demand forecasting) let the model learn from future patterns.
• Group leakage: the same customer/patient appears in both train and test, so the model memorizes individual-specific signals.

Why it matters and how practitioners prevent it

Leakage leads to overconfident model selection, wrong hyperparameters, and nasty surprises after deployment. The practical fix is to make the split reflect reality (time-based or group-based when needed) and to fit all preprocessing only on the training fold using a pipeline. In scikit-learn, using Pipeline ensures steps like StandardScaler are fit inside cross-validation:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
  ("scaler", StandardScaler()),
  ("model", LogisticRegression())
])