Skip to content
Kuza Ndoto Academy

Random Forests

IntermediateChapter 3 · Page 13 of 264 min readUpdated 15 Nov 2026

A random forest combines the predictions of many decision trees, each trained on a slightly different version of the data. One tree can be unreliable, but the "wisdom of the crowd" of hundreds of trees is usually accurate and stable. Random forests are among the best all-round models for tabular data.

Why many trees beat one

A single decision tree is unstable: change a few training rows and you may get a very different tree. Each tree makes mistakes, but if the trees are different from one another, their mistakes tend to point in different directions and cancel out when you combine them. A random forest deliberately makes its trees different in two ways.

Ingredient 1: bootstrap samples (bagging)

Each tree is trained on a bootstrap sample: a random sample of the training rows, of the same size, drawn with replacement. Some rows appear twice or more; others not at all. On average a bootstrap sample contains about 63% of the distinct original rows (the exact limit is 1 − 1/e ≈ 0.632). Training models on bootstrap samples and combining them is called bagging (bootstrap aggregating).

The rows a tree didn't see, roughly 37%, are its out-of-bag (OOB) rows. They act like a free validation set: each row can be scored using only the trees that never trained on it.

Ingredient 2: random feature subsets

At every split, a tree may only consider a random subset of the features. For classification, scikit-learn's default is the square root of the number of features (max_features="sqrt"). Without this, every tree would pick the same strongest feature at the top and the trees would be too similar.

Combining the trees

  • Classification: each tree votes, and the forest predicts the majority. (scikit-learn actually averages the trees' predicted probabilities, which usually gives the same result.) If 5 trees say "fraud", "fraud", "legit", "fraud", "legit", the forest says fraud, 3 votes to 2.
  • Regression: the forest predicts the average of the trees' predictions. If three trees predict a rent of 20,000, 24,000 and 22,000 KES, the forest predicts 22,000.
new transaction fraud fraud legit fraud legit Majority vote: fraud (3 to 2)
Figure 1. Each tree was trained on its own bootstrap sample with random feature choices, so the trees disagree. The forest combines them; here it predicts fraud.

Random forest in scikit-learn

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)

forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
forest.fit(X_train, y_train)

print(f"Test accuracy: {forest.score(X_test, y_test):.2f}")
print(f"OOB score: {forest.oob_score_:.2f}")

ranked = sorted(zip(iris.feature_names, forest.feature_importances_),
                key=lambda pair: pair[1], reverse=True)
for name, importance in ranked:
    print(f"{name:20} {importance:.2f}")

You should see a test accuracy and OOB score of roughly 0.93 to 1.00 (iris is an easy dataset). In the importance ranking, the two petal measurements should come out on top with most of the importance between them, and sepal width at the bottom. Exact numbers depend on your library version, so treat them as indicative.

Note

feature_importances_ measures how much each feature reduced impurity across all trees. It's a quick guide, but it can overrate features with many distinct values. For a more reliable check use sklearn.inspection.permutation_importance on the test set.

Important settings

ParameterMeaningTip
n_estimatorsNumber of treesMore is rarely worse, only slower; 100–500 is typical
max_depthMaximum depth of each treeDefault None (fully grown) often works well
max_featuresFeatures considered per splitSmaller values make trees more different
min_samples_leafMinimum rows per leafRaise it to smooth predictions on noisy data
class_weightWeight per class"balanced" helps with rare classes
n_jobsCPU cores to use-1 uses all cores

Strengths and weaknesses

  • Strengths: strong accuracy with little tuning, no scaling needed, handles non-linear patterns, much less overfitting than a single tree, and gives feature importances.
  • Weaknesses: slower and larger than one tree, harder to explain (you can't draw 200 trees), and like all trees it can't extrapolate beyond the range of the training labels in regression.

Random forests are a member of the ensemble family. Another popular ensemble is gradient boosting (for example HistGradientBoostingClassifier, XGBoost or LightGBM), which builds trees one after another, each correcting the previous ones' errors.

Common mistake

Assuming a forest can't overfit, so evaluation can be skipped. Forests resist overfitting well, but data leakage, duplicated rows across train and test, or a tiny dataset can still produce misleading scores. Always evaluate on held-out data.

Key points

  • A random forest averages many decision trees trained on bootstrap samples.
  • Random feature subsets at each split make the trees diverse.
  • Classification uses a vote (averaged probabilities); regression uses the mean.
  • The OOB score gives a free estimate of performance on unseen data.
Up next · New chapter: Unsupervised LearningK-Means ClusteringGroup similar data automatically.
→

Tip: use the ← and → keys to move between pages.