Random Forests
A random forest combines the predictions of many decision trees, each trained on a slightly different version of the data. One tree can be unreliable, but the "wisdom of the crowd" of hundreds of trees is usually accurate and stable. Random forests are among the best all-round models for tabular data.
Why many trees beat one
A single decision tree is unstable: change a few training rows and you may get a very different tree. Each tree makes mistakes, but if the trees are different from one another, their mistakes tend to point in different directions and cancel out when you combine them. A random forest deliberately makes its trees different in two ways.
Ingredient 1: bootstrap samples (bagging)
Each tree is trained on a bootstrap sample: a random sample of the training rows, of the same size, drawn with replacement. Some rows appear twice or more; others not at all. On average a bootstrap sample contains about 63% of the distinct original rows (the exact limit is 1 − 1/e ≈ 0.632). Training models on bootstrap samples and combining them is called bagging (bootstrap aggregating).
The rows a tree didn't see, roughly 37%, are its out-of-bag (OOB) rows. They act like a free validation set: each row can be scored using only the trees that never trained on it.
Ingredient 2: random feature subsets
At every split, a tree may only consider a random subset of the features. For classification, scikit-learn's default is the square root of the number of features (max_features="sqrt"). Without this, every tree would pick the same strongest feature at the top and the trees would be too similar.
Combining the trees
- Classification: each tree votes, and the forest predicts the majority. (scikit-learn actually averages the trees' predicted probabilities, which usually gives the same result.) If 5 trees say "fraud", "fraud", "legit", "fraud", "legit", the forest says fraud, 3 votes to 2.
- Regression: the forest predicts the average of the trees' predictions. If three trees predict a rent of 20,000, 24,000 and 22,000 KES, the forest predicts 22,000.
Random forest in scikit-learn
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, test_size=0.2, random_state=42, stratify=iris.target
)
forest = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
forest.fit(X_train, y_train)
print(f"Test accuracy: {forest.score(X_test, y_test):.2f}")
print(f"OOB score: {forest.oob_score_:.2f}")
ranked = sorted(zip(iris.feature_names, forest.feature_importances_),
key=lambda pair: pair[1], reverse=True)
for name, importance in ranked:
print(f"{name:20} {importance:.2f}")
You should see a test accuracy and OOB score of roughly 0.93 to 1.00 (iris is an easy dataset). In the importance ranking, the two petal measurements should come out on top with most of the importance between them, and sepal width at the bottom. Exact numbers depend on your library version, so treat them as indicative.
feature_importances_ measures how much each feature reduced impurity across all trees. It's a quick guide, but it can overrate features with many distinct values. For a more reliable check use sklearn.inspection.permutation_importance on the test set.
Important settings
| Parameter | Meaning | Tip |
|---|---|---|
n_estimators | Number of trees | More is rarely worse, only slower; 100–500 is typical |
max_depth | Maximum depth of each tree | Default None (fully grown) often works well |
max_features | Features considered per split | Smaller values make trees more different |
min_samples_leaf | Minimum rows per leaf | Raise it to smooth predictions on noisy data |
class_weight | Weight per class | "balanced" helps with rare classes |
n_jobs | CPU cores to use | -1 uses all cores |
Strengths and weaknesses
- Strengths: strong accuracy with little tuning, no scaling needed, handles non-linear patterns, much less overfitting than a single tree, and gives feature importances.
- Weaknesses: slower and larger than one tree, harder to explain (you can't draw 200 trees), and like all trees it can't extrapolate beyond the range of the training labels in regression.
Random forests are a member of the ensemble family. Another popular ensemble is gradient boosting (for example HistGradientBoostingClassifier, XGBoost or LightGBM), which builds trees one after another, each correcting the previous ones' errors.
Assuming a forest can't overfit, so evaluation can be skipped. Forests resist overfitting well, but data leakage, duplicated rows across train and test, or a tiny dataset can still produce misleading scores. Always evaluate on held-out data.
Key points
- A random forest averages many decision trees trained on bootstrap samples.
- Random feature subsets at each split make the trees diverse.
- Classification uses a vote (averaged probabilities); regression uses the mean.
- The OOB score gives a free estimate of performance on unseen data.
Tip: use the ← and → keys to move between pages.
