Notes
🔍

Gini Impurity

When a decision tree chooses a split, it’s really asking: “If I separate the data this way, do the groups become more clear-cut?” Gini impurity is a simple score that answers that question by measuring how mixed the class labels are at a node.

What Gini impurity measures

For a classification node with class proportions \(p_1, p_2, \dots, p_K\), Gini impurity is:

Gini = 1 - Σ(p_k^2)

It’s the probability of mislabeling a random example if you assign a label by sampling according to the node’s class distribution. A node is “pure” (all one class) when Gini = 0. With two classes, the maximum is 0.5 at a 50/50 mix; with more classes, the maximum increases but still reflects “more mixing = higher impurity.”

How trees use it to pick splits

A decision tree evaluates candidate splits and computes the weighted average impurity of the child nodes. The best split is the one with the largest reduction in impurity (often called Gini gain):

  • Parent impurity: how mixed the node is before splitting
  • Children impurity: how mixed the left/right nodes are after splitting, weighted by their sizes
  • Choose the split that makes the children as pure as possible

In scikit-learn, this is the default criterion for 

DecisionTreeClassifier(criterion="gini")
.

Why it matters in real problems

In spam detection or fraud detection, good early splits quickly isolate “obvious” spam/fraud patterns, creating purer nodes and simpler downstream decisions. If impurity reduction is weak (for example, due to noisy features), trees can grow deep trying to chase purity, increasing overfitting—one reason pruning and depth limits matter.

Gini Impurity is a node-level measure of class heterogeneity in a classification decision tree, defined as 1 − Σ p(k)² over classes k, where p(k) is the class proportion at the node. It equals 0 for a pure node and increases as classes mix. It matters because tree-splitting criteria minimize Gini Impurity to choose splits that best separate classes, directly shaping tree structure and predictive performance.

Imagine you have a jar of mixed candies: some are red, some are blue. If the jar is all one color, it’s easy to guess what you’ll grab next. If it’s a messy mix, guessing is harder. Gini Impurity is a simple way to describe how “mixed up” a group is.

In a decision tree, the model keeps splitting data into smaller groups to make clearer decisions, like separating candies by color. Gini Impurity helps the tree choose splits that make each group as “pure” as possible—mostly one class—so the final predictions are more confident and consistent.