Splitting Criterion

When a decision tree grows, it keeps asking: “What question should I ask next to best separate the data?” The splitting criterion is the scoring rule that answers that question by measuring how “good” a candidate split is.

What it is and how it works

At each node, a tree considers many possible splits (feature and threshold, like age < 35). For each candidate, the splitting criterion computes how much the split improves the node’s “purity” (classification) or reduces prediction error (regression). The tree picks the split with the best score, then repeats this process on the child nodes. Conceptually, it’s like choosing the next yes/no question that most reduces uncertainty about the target.

Common criteria you’ll see

Gini impurity (classification): prefers splits that create child nodes dominated by a single class.
Entropy / Information Gain (classification): chooses splits that maximize the reduction in label uncertainty.
Variance reduction / MSE (regression): chooses splits that most reduce squared error within child nodes.

In scikit-learn’s DecisionTreeClassifier, this is controlled by the criterion parameter (e.g., "gini" or "entropy" / "log_loss"), and for regression by DecisionTreeRegressor(criterion="squared_error").

Why it matters in practice

The criterion shapes the tree’s behavior: which features get used early, how sensitive the tree is to class imbalance, and how quickly it overfits noisy patterns. For example, in spam detection, a good split might isolate messages containing certain tokens; in credit scoring, it might separate applicants by debt-to-income thresholds. If the criterion doesn’t match the goal (e.g., using a regression-style criterion for classification), the tree makes systematically weaker splits and accuracy drops.

A splitting criterion is the objective function a decision tree optimizes to choose the best feature and threshold at each node, quantifying how much a candidate split improves label purity or reduces prediction error. Common splitting criteria include Gini impurity and information gain for classification, and variance/MSE reduction for regression. It matters because it directly determines the tree’s structure, accuracy, and generalization behavior.

Think of sorting a big pile of mail. You could split it by “local vs. out of town,” or by “personal vs. business.” A splitting criterion is the rule a decision tree uses to choose which question to ask next so the pile becomes as neatly separated as possible.

In supervised learning, a decision tree learns from examples with known answers (like “spam” or “not spam”). At each step, it tries different questions about the data (like “Does the email contain ‘free’?”) and uses a splitting criterion to pick the one that best separates the correct labels, making later predictions more reliable.