Skip to content
Kuza Ndoto Academy

Overfitting & Underfitting

IntermediateChapter 5 · Page 17 of 264 min readUpdated 15 Nov 2026

A model overfits when it memorises the training data, noise included, and then performs poorly on new data. It underfits when it's too simple to capture the real pattern. Finding the balance between the two is the central challenge of machine learning.

An exam analogy

  • Underfitting: a student who barely studied. They do badly on past papers and on the real exam.
  • Overfitting: a student who memorised the answers to last year's paper word for word. They score 100% on that paper but fail when the questions change.
  • A good fit: a student who understood the concepts. They do well on past papers and on new questions.

How to spot each one

Compare the score on training data with the score on validation or test data:

Training scoreTest scoreDiagnosis
UnderfittingLowLowModel too simple (high bias)
OverfittingVery highMuch lowerModel too complex (high variance)
Good fitHighClose to trainingAbout right

Bias is error from wrong assumptions, such as fitting a straight line to a curve. Variance is error from being too sensitive to the particular training sample; train on a slightly different sample and the model changes a lot. Making a model more complex lowers bias but raises variance. This is the bias-variance trade-off.

Model complexity (e.g. tree depth) Error Underfitting Overfitting sweet spot validation error training error
Figure 1. A schematic (not real data). Training error keeps falling as the model gets more complex, but validation error turns upward once the model starts memorising noise. The gap on the right is overfitting.

Seeing it in code

A decision tree's max_depth controls its complexity. Let's compare three depths on the built-in breast cancer dataset (569 tumour samples, 30 features):

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

for depth in [1, 3, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    tree.fit(X_train, y_train)
    print(depth, f"train={tree.score(X_train, y_train):.3f}",
          f"test={tree.score(X_test, y_test):.3f}")

Exact numbers vary with library version, but you should see this pattern:

  • Depth 1 (one question): training and test accuracy both around 0.9. It's simple, and slightly underfit.
  • Depth 3: both scores rise, and the gap between them is small.
  • Depth None (unlimited): training accuracy of 1.000, because the tree keeps splitting until every leaf is pure, while test accuracy is typically no better than depth 3 and often worse. The gap is the signature of overfitting.

A picture in words

Imagine 10 points that roughly follow a gentle curve. A straight line misses the curve: underfitting. A smooth curve follows the trend: a good fit. A wiggly line of degree 9 can pass through all 10 points exactly, so training error is zero, but it swings wildly between them and makes absurd predictions for new points. More flexibility is not always better.

How to fix overfitting

  • Get more training data. Memorising becomes harder and the true pattern clearer.
  • Simplify the model: limit tree depth, use fewer features, or pick a less flexible algorithm.
  • Regularisation: add a penalty for large weights. Ridge (L2) and Lasso (L1) do this for linear models; in LogisticRegression, smaller C means stronger regularisation.
  • Ensembles: averaging many trees, as in random forests, cuts variance.
  • For neural networks: dropout, early stopping (stop training when validation loss starts rising) and data augmentation.
  • Feature selection: remove noisy or irrelevant features.

How to fix underfitting

  • Use a more flexible model (deeper trees, a forest, a neural network).
  • Add better features through feature engineering.
  • Reduce regularisation.
  • Train for longer (for models trained iteratively).
Tip

sklearn.model_selection.learning_curve trains a model on growing amounts of data and reports training and validation scores. If the two curves are far apart, more data will probably help (overfitting). If they meet at a low score, more data won't help; you need a better model or features (underfitting).

Common mistake

Overfitting to the test set. If you try fifty models and keep the one with the best test score, you've tuned to that particular test set, and its score is now too optimistic. Choose models with cross-validation on training data, and keep the test set for one final check.

Common mistake

Mistaking data leakage for a great model. If test accuracy is suspiciously close to 100% on a hard problem, look for leakage (a feature that contains the answer, duplicates across train and test, or preprocessing fitted on all the data) before celebrating.

Key points

  • Overfitting: great on training data, poor on new data. Underfitting: poor on both.
  • Compare training and validation scores to diagnose which you have.
  • Complexity trades bias against variance.
  • More data, simpler models, regularisation and ensembles reduce overfitting.
Up next · Model EvaluationCross-ValidationMore reliable model testing.
→

Tip: use the ← and → keys to move between pages.