Decision Trees
A decision tree makes predictions by asking a series of yes/no questions about the features, like a flowchart: "Is income above KES 37,500? Has this person defaulted before?" Trees are easy to visualise and explain, handle numbers and categories, and are the building block of powerful models such as random forests.
How a tree makes a prediction
A tree starts at the root node, which asks a question about one feature. Depending on the answer, you follow a branch to the next node, and so on until you reach a leaf, which gives the prediction. For classification, a leaf predicts the majority class of the training examples that ended up there; for regression, it predicts their average value.
How a tree chooses its questions: Gini impurity
When training, the tree looks at every feature and every possible threshold and picks the split that makes the resulting groups as pure as possible, meaning each group is mostly one class. scikit-learn measures impurity with the Gini index by default:
Gini = 1 − (p₁² + p₂² + …), where p is the proportion of each class in the node.
- A pure node (all "yes"): Gini = 1 − 1² = 0.
- A 50/50 node: Gini = 1 − (0.5² + 0.5²) = 0.5, the worst possible for two classes.
- 4 "yes" and 6 "no": Gini = 1 − (0.4² + 0.6²) = 1 − 0.52 = 0.48.
A split is scored by the weighted average Gini of its child nodes. The tree picks the split with the lowest score, then repeats the process inside each child. An alternative measure, entropy (criterion="entropy"), usually picks very similar splits.
A worked example: loan approval
Six past loan applications, with monthly income in thousands of KES and whether the applicant had defaulted before:
| Income | Has defaulted | Approved? |
|---|---|---|
| 20 | 0 | no |
| 25 | 0 | no |
| 50 | 0 | yes |
| 60 | 0 | yes |
| 70 | 1 | no |
| 80 | 0 | yes |
The root holds 3 yes and 3 no, so its Gini is 0.5. Compare two candidate splits:
- has_defaulted <= 0.5: left gets 5 rows (3 yes, 2 no, Gini 0.48); right gets 1 row (pure, Gini 0). Weighted: (5/6) × 0.48 + (1/6) × 0 = 0.40.
- income <= 37.5: left gets 2 rows (both no, Gini 0); right gets 4 rows (3 yes, 1 no, Gini 1 − (0.75² + 0.25²) = 0.375). Weighted: (2/6) × 0 + (4/6) × 0.375 = 0.25.
Checking every other threshold gives nothing lower than 0.25, so income <= 37.5 becomes the root question. (Thresholds are placed halfway between neighbouring values: 37.5 is between 25 and 50.) Inside the right branch, "has_defaulted" separates the remaining rows perfectly. scikit-learn finds the same tree:
from sklearn.tree import DecisionTreeClassifier, export_text X = [[20, 0], [25, 0], [50, 0], [60, 0], [70, 1], [80, 0]] y = ["no", "no", "yes", "yes", "no", "yes"] tree = DecisionTreeClassifier(max_depth=2, random_state=42) tree.fit(X, y) print(export_text(tree, feature_names=["income", "has_defaulted"]))
print(tree.predict([[45, 0], [90, 1]]))
An applicant earning 45 with no default is approved; one earning 90 who has defaulted is not. You can read and explain every decision, which matters in areas like lending.
export_text, with the Gini impurity of each node. Every leaf is pure (Gini 0).Overfitting and pruning
Left alone, a tree keeps splitting until every leaf is pure. It can end up with one leaf per training example: perfect training accuracy and poor performance on new data. This is overfitting. Control tree size with these settings:
max_depth: the maximum number of questions from root to leaf.min_samples_leaf: each leaf must contain at least this many training examples.min_samples_split: a node needs this many examples before it may split.ccp_alpha: cost-complexity pruning, which removes branches that add little.
Choose these values with cross-validation, not by guessing.
Reporting a deep tree's training accuracy of 100% as if it were real performance. Always evaluate on a held-out test set. A tree that is perfect on training data is a warning sign, not a success.
Strengths and weaknesses
| Strengths | Weaknesses |
|---|---|
| Easy to explain and draw | Overfits easily if not limited |
| No feature scaling needed | Unstable: small changes in data can produce a very different tree |
| Captures non-linear patterns and interactions | Boundaries are made of axis-aligned steps |
Works for classification and regression (DecisionTreeRegressor) | Usually less accurate than an ensemble of trees |
The weaknesses are largely fixed by combining many trees: see Random Forests.
Key points
- A decision tree predicts by following yes/no questions from root to leaf.
- Splits are chosen to reduce impurity; Gini = 1 − the sum of squared class proportions.
- Trees need no scaling and are easy to explain.
- Limit depth or leaf size to prevent overfitting.
Tip: use the ← and → keys to move between pages.
