Train-Validation-Test Split

When you build a predictive model, it’s tempting to judge it on the same data you used to teach it. The problem is that models can “memorize” patterns that won’t hold up in the real world, so you need a fair way to check how well they generalize.

What the split is and what each part does

A train-validation-test split partitions labeled data into three disjoint sets:

• Training set: used to fit model parameters (e.g., the weights in linear/logistic regression, tree splits in random forests).
• Validation set: used during development to choose hyperparameters and make modeling decisions (feature choices, regularization strength, early stopping, model selection).
• Test set: used once at the end to estimate real-world performance. It acts like a “final exam” and should not influence any choices.

Why it matters (and what goes wrong without it)

If you tune on the test set—even indirectly—you get an overly optimistic score because you’ve adapted to that specific dataset. This is a form of data leakage. The split also forces you to separate “learning” from “evaluating,” which makes metrics like accuracy, AUC, RMSE, or MAE meaningful indicators of future behavior.

Practical patterns you’ll see

• Spam detection: train on past labeled emails, validate to pick thresholds and regularization, test to estimate deployment accuracy.
• Credit scoring: use a stratified split to preserve default rates across sets.
• Demand forecasting: use a time-based split (train on earlier dates, validate on later, test on the most recent) to match how the model will be used.

from sklearn.model_selection import train_test_split

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)