Notes
🔍

Information Gain

When a decision tree “asks a question” like “Is the email subject line all caps?”, it’s trying to pick the question that most quickly clears up uncertainty about the label. Information Gain is the score that tells the tree how much a candidate split reduces that uncertainty.

What it measures

In classification trees, uncertainty is measured with entropy, which is highest when classes are evenly mixed and lowest when a node is pure (all one class). Information Gain for a split is:

  • IG = Entropy(parent) − Weighted average Entropy(children)

The “weighted average” part matters because a split that creates one huge child node and one tiny child node should mostly be judged by what happens in the huge node. A high information gain means the split produces children that are much more class-pure than the parent.

How it’s used in a tree

At each node, the algorithm evaluates many possible splits (different features and thresholds) and chooses the one with the largest information gain. For example:

  • Spam detection: splitting on “contains ‘free’” might sharply separate spam from non-spam, giving high gain.
  • Disease diagnosis: a lab value threshold that separates positive from negative cases reduces uncertainty and earns high gain.
  • Customer churn: “contract type” might create groups with very different churn rates, increasing purity.

Why it matters (and a key caveat)

Information gain is what makes trees greedy-but-effective: each split is chosen to be maximally informative right now. The caveat is that it can favor features with many distinct values (they can create very pure-looking partitions). Many implementations address this with alternatives like gain ratio (C4.5) or by using Gini impurity (common in scikit-learn’s trees) instead of entropy.

Information Gain is the reduction in target uncertainty (entropy) achieved by splitting a dataset on a feature, computed as the parent node’s entropy minus the weighted average entropy of the child nodes. It quantifies how well a split separates the labels (e.g., in classification trees). Information Gain matters because it drives split selection in many decision-tree learners, directly affecting tree structure, predictive accuracy, and generalization.

Imagine you’re playing “20 Questions.” Some questions are great because they quickly narrow down the answer, while others barely help. Information Gain is the AI version of that idea: it measures how much a question (a data feature like “Is the email from your contacts?”) helps reduce confusion.

In a decision tree, the model keeps asking simple yes/no-style questions to sort examples into groups, like spam vs. not spam or sick vs. healthy. Information Gain helps the tree pick the next question that makes the groups “cleaner” and more certain, so the final decision is easier and more accurate.