Skip to content
Kuza Ndoto Academy

Train/Test Split

IntermediateChapter 2 · Page 8 of 263 min readUpdated 15 Nov 2026

A model that is tested on the same data it learned from will always look better than it really is, like a student who saw the exam paper in advance. A train/test split keeps some data aside so you can measure how well the model handles examples it has never seen.

Why you need a test set

The goal of machine learning is generalisation: performing well on new data. A flexible model can memorise its training data perfectly, including the noise, and still fail on new cases. This is overfitting. Only data the model hasn't trained on can reveal it.

  • Training set (usually 70–80%): the model learns from this.
  • Test set (usually 20–30%): used once, at the end, for an honest score.

train_test_split in scikit-learn

import numpy as np
from sklearn.model_selection import train_test_split

X = np.arange(40).reshape(20, 2)        # 20 samples, 2 features
y = np.array([0] * 15 + [1] * 5)        # 15 of class 0, 5 of class 1

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(X_train.shape, X_test.shape)
print(np.bincount(y_train), np.bincount(y_test))
(16, 2) (4, 2) [12 4] [3 1]
80% training 20% test X_train, y_train: 16 rows X_test: 4 rows 12 of class 0 + 4 of class 1 3 + 1 each square is one row; stratify=y keeps 25% class 1 in both sets
Figure 1. The split from the code: 16 rows for training and 4 for testing, with class proportions preserved by stratify=y.

What each argument does:

  • test_size=0.2: 20% of the 20 rows (4 rows) go to the test set.
  • random_state=42: rows are shuffled before splitting; fixing the seed makes the shuffle the same every run, so results are reproducible. Any number works; 42 is just a popular choice.
  • stratify=y: keeps the class proportions the same in both sets. The full data is 75% class 0 and 25% class 1, and so is each part: 12 and 4 in training, 3 and 1 in testing.
Tip

Always use stratify=y for classification, especially when one class is rare. Without it, a random split of a dataset with few fraud cases could put almost none of them in the test set.

Training, validation and test sets

While building a model you make many choices: which algorithm, how deep a tree, how many neighbours. If you make those choices by checking the test score, the test set stops being "unseen". The fix is a third set, the validation set, used for comparing options. A common split is 60% train, 20% validation, 20% test:

X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)
# 0.25 of the remaining 80% is 20% of the original data

With small datasets a single validation set is noisy; cross-validation is the better tool for model selection.

Special cases

Time series

If you're predicting the future (next month's sales, tomorrow's matatu demand), never shuffle. Train on the past and test on the most recent period, otherwise the model "sees the future" during training. Use train_test_split(..., shuffle=False) or TimeSeriesSplit.

Groups

If one customer, patient or farm appears in many rows, keep all of their rows on the same side of the split (see GroupShuffleSplit). Otherwise the model can recognise the individual rather than learn the general pattern.

Common mistake

Preprocessing before splitting. Scaling, imputing missing values or selecting features on the whole dataset and then splitting lets test information leak into training. The right order is: split first, then fit every preprocessing step on the training set only. Pipelines make this automatic.

Common mistake

Duplicate rows that end up in both sets. The model gets "test" questions it has literally memorised. Remove duplicates before splitting.

Key points

  • Hold back a test set to measure generalisation honestly.
  • Use random_state for reproducibility and stratify=y for classification.
  • Use a validation set or cross-validation for choices; use the test set once.
  • Don't shuffle time series; keep groups together; split before preprocessing.
Up next · New chapter: Supervised LearningLinear RegressionPredict numbers with a straight line.
→

Tip: use the ← and → keys to move between pages.