Skip to content
Kuza Ndoto Academy

Decision Trees

IntermediateChapter 3 · Page 11 of 264 min readUpdated 15 Nov 2026

A decision tree makes predictions by asking a series of yes/no questions about the features, like a flowchart: "Is income above KES 37,500? Has this person defaulted before?" Trees are easy to visualise and explain, handle numbers and categories, and are the building block of powerful models such as random forests.

How a tree makes a prediction

A tree starts at the root node, which asks a question about one feature. Depending on the answer, you follow a branch to the next node, and so on until you reach a leaf, which gives the prediction. For classification, a leaf predicts the majority class of the training examples that ended up there; for regression, it predicts their average value.

How a tree chooses its questions: Gini impurity

When training, the tree looks at every feature and every possible threshold and picks the split that makes the resulting groups as pure as possible, meaning each group is mostly one class. scikit-learn measures impurity with the Gini index by default:

Gini = 1 − (p₁² + p₂² + …), where p is the proportion of each class in the node.

  • A pure node (all "yes"): Gini = 1 − 1² = 0.
  • A 50/50 node: Gini = 1 − (0.5² + 0.5²) = 0.5, the worst possible for two classes.
  • 4 "yes" and 6 "no": Gini = 1 − (0.4² + 0.6²) = 1 − 0.52 = 0.48.

A split is scored by the weighted average Gini of its child nodes. The tree picks the split with the lowest score, then repeats the process inside each child. An alternative measure, entropy (criterion="entropy"), usually picks very similar splits.

A worked example: loan approval

Six past loan applications, with monthly income in thousands of KES and whether the applicant had defaulted before:

IncomeHas defaultedApproved?
200no
250no
500yes
600yes
701no
800yes

The root holds 3 yes and 3 no, so its Gini is 0.5. Compare two candidate splits:

  • has_defaulted <= 0.5: left gets 5 rows (3 yes, 2 no, Gini 0.48); right gets 1 row (pure, Gini 0). Weighted: (5/6) × 0.48 + (1/6) × 0 = 0.40.
  • income <= 37.5: left gets 2 rows (both no, Gini 0); right gets 4 rows (3 yes, 1 no, Gini 1 − (0.75² + 0.25²) = 0.375). Weighted: (2/6) × 0 + (4/6) × 0.375 = 0.25.

Checking every other threshold gives nothing lower than 0.25, so income <= 37.5 becomes the root question. (Thresholds are placed halfway between neighbouring values: 37.5 is between 25 and 50.) Inside the right branch, "has_defaulted" separates the remaining rows perfectly. scikit-learn finds the same tree:

from sklearn.tree import DecisionTreeClassifier, export_text

X = [[20, 0], [25, 0], [50, 0], [60, 0], [70, 1], [80, 0]]
y = ["no", "no", "yes", "yes", "no", "yes"]

tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(X, y)
print(export_text(tree, feature_names=["income", "has_defaulted"]))
|--- income <= 37.50 | |--- class: no |--- income > 37.50 | |--- has_defaulted <= 0.50 | | |--- class: yes | |--- has_defaulted > 0.50 | | |--- class: no
print(tree.predict([[45, 0], [90, 1]]))
['yes' 'no']

An applicant earning 45 with no default is approved; one earning 90 who has defaulted is not. You can read and explain every decision, which matters in areas like lending.

TrueFalse TrueFalse income <= 37.5? gini 0.5, 3 yes / 3 no class: no gini 0, 2 rows has_defaulted <= 0.5? gini 0.375, 3 yes / 1 no class: yes gini 0, 3 rows class: no gini 0, 1 row
Figure 1. The tree printed by export_text, with the Gini impurity of each node. Every leaf is pure (Gini 0).

Overfitting and pruning

Left alone, a tree keeps splitting until every leaf is pure. It can end up with one leaf per training example: perfect training accuracy and poor performance on new data. This is overfitting. Control tree size with these settings:

  • max_depth: the maximum number of questions from root to leaf.
  • min_samples_leaf: each leaf must contain at least this many training examples.
  • min_samples_split: a node needs this many examples before it may split.
  • ccp_alpha: cost-complexity pruning, which removes branches that add little.

Choose these values with cross-validation, not by guessing.

Common mistake

Reporting a deep tree's training accuracy of 100% as if it were real performance. Always evaluate on a held-out test set. A tree that is perfect on training data is a warning sign, not a success.

Strengths and weaknesses

StrengthsWeaknesses
Easy to explain and drawOverfits easily if not limited
No feature scaling neededUnstable: small changes in data can produce a very different tree
Captures non-linear patterns and interactionsBoundaries are made of axis-aligned steps
Works for classification and regression (DecisionTreeRegressor)Usually less accurate than an ensemble of trees

The weaknesses are largely fixed by combining many trees: see Random Forests.

Key points

  • A decision tree predicts by following yes/no questions from root to leaf.
  • Splits are chosen to reduce impurity; Gini = 1 − the sum of squared class proportions.
  • Trees need no scaling and are easy to explain.
  • Limit depth or leaf size to prevent overfitting.
Up next · Supervised LearningK-Nearest NeighboursPredict from the most similar examples.
→

Tip: use the ← and → keys to move between pages.