Overfitting & Underfitting
A model overfits when it memorises the training data, noise included, and then performs poorly on new data. It underfits when it's too simple to capture the real pattern. Finding the balance between the two is the central challenge of machine learning.
An exam analogy
- Underfitting: a student who barely studied. They do badly on past papers and on the real exam.
- Overfitting: a student who memorised the answers to last year's paper word for word. They score 100% on that paper but fail when the questions change.
- A good fit: a student who understood the concepts. They do well on past papers and on new questions.
How to spot each one
Compare the score on training data with the score on validation or test data:
| Training score | Test score | Diagnosis | |
|---|---|---|---|
| Underfitting | Low | Low | Model too simple (high bias) |
| Overfitting | Very high | Much lower | Model too complex (high variance) |
| Good fit | High | Close to training | About right |
Bias is error from wrong assumptions, such as fitting a straight line to a curve. Variance is error from being too sensitive to the particular training sample; train on a slightly different sample and the model changes a lot. Making a model more complex lowers bias but raises variance. This is the bias-variance trade-off.
Seeing it in code
A decision tree's max_depth controls its complexity. Let's compare three depths on the built-in breast cancer dataset (569 tumour samples, 30 features):
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y
)
for depth in [1, 3, None]:
tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
tree.fit(X_train, y_train)
print(depth, f"train={tree.score(X_train, y_train):.3f}",
f"test={tree.score(X_test, y_test):.3f}")
Exact numbers vary with library version, but you should see this pattern:
- Depth 1 (one question): training and test accuracy both around 0.9. It's simple, and slightly underfit.
- Depth 3: both scores rise, and the gap between them is small.
- Depth None (unlimited): training accuracy of 1.000, because the tree keeps splitting until every leaf is pure, while test accuracy is typically no better than depth 3 and often worse. The gap is the signature of overfitting.
A picture in words
Imagine 10 points that roughly follow a gentle curve. A straight line misses the curve: underfitting. A smooth curve follows the trend: a good fit. A wiggly line of degree 9 can pass through all 10 points exactly, so training error is zero, but it swings wildly between them and makes absurd predictions for new points. More flexibility is not always better.
How to fix overfitting
- Get more training data. Memorising becomes harder and the true pattern clearer.
- Simplify the model: limit tree depth, use fewer features, or pick a less flexible algorithm.
- Regularisation: add a penalty for large weights.
Ridge(L2) andLasso(L1) do this for linear models; inLogisticRegression, smallerCmeans stronger regularisation. - Ensembles: averaging many trees, as in random forests, cuts variance.
- For neural networks: dropout, early stopping (stop training when validation loss starts rising) and data augmentation.
- Feature selection: remove noisy or irrelevant features.
How to fix underfitting
- Use a more flexible model (deeper trees, a forest, a neural network).
- Add better features through feature engineering.
- Reduce regularisation.
- Train for longer (for models trained iteratively).
sklearn.model_selection.learning_curve trains a model on growing amounts of data and reports training and validation scores. If the two curves are far apart, more data will probably help (overfitting). If they meet at a low score, more data won't help; you need a better model or features (underfitting).
Overfitting to the test set. If you try fifty models and keep the one with the best test score, you've tuned to that particular test set, and its score is now too optimistic. Choose models with cross-validation on training data, and keep the test set for one final check.
Mistaking data leakage for a great model. If test accuracy is suspiciously close to 100% on a hard problem, look for leakage (a feature that contains the answer, duplicates across train and test, or preprocessing fitted on all the data) before celebrating.
Key points
- Overfitting: great on training data, poor on new data. Underfitting: poor on both.
- Compare training and validation scores to diagnose which you have.
- Complexity trades bias against variance.
- More data, simpler models, regularisation and ensembles reduce overfitting.
Tip: use the ← and → keys to move between pages.
