Train/Test Split
A model that is tested on the same data it learned from will always look better than it really is, like a student who saw the exam paper in advance. A train/test split keeps some data aside so you can measure how well the model handles examples it has never seen.
Why you need a test set
The goal of machine learning is generalisation: performing well on new data. A flexible model can memorise its training data perfectly, including the noise, and still fail on new cases. This is overfitting. Only data the model hasn't trained on can reveal it.
- Training set (usually 70–80%): the model learns from this.
- Test set (usually 20–30%): used once, at the end, for an honest score.
train_test_split in scikit-learn
import numpy as np
from sklearn.model_selection import train_test_split
X = np.arange(40).reshape(20, 2) # 20 samples, 2 features
y = np.array([0] * 15 + [1] * 5) # 15 of class 0, 5 of class 1
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
print(X_train.shape, X_test.shape)
print(np.bincount(y_train), np.bincount(y_test))
stratify=y.What each argument does:
test_size=0.2: 20% of the 20 rows (4 rows) go to the test set.random_state=42: rows are shuffled before splitting; fixing the seed makes the shuffle the same every run, so results are reproducible. Any number works; 42 is just a popular choice.stratify=y: keeps the class proportions the same in both sets. The full data is 75% class 0 and 25% class 1, and so is each part: 12 and 4 in training, 3 and 1 in testing.
Always use stratify=y for classification, especially when one class is rare. Without it, a random split of a dataset with few fraud cases could put almost none of them in the test set.
Training, validation and test sets
While building a model you make many choices: which algorithm, how deep a tree, how many neighbours. If you make those choices by checking the test score, the test set stops being "unseen". The fix is a third set, the validation set, used for comparing options. A common split is 60% train, 20% validation, 20% test:
X_temp, X_test, y_temp, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)
# 0.25 of the remaining 80% is 20% of the original data
With small datasets a single validation set is noisy; cross-validation is the better tool for model selection.
Special cases
Time series
If you're predicting the future (next month's sales, tomorrow's matatu demand), never shuffle. Train on the past and test on the most recent period, otherwise the model "sees the future" during training. Use train_test_split(..., shuffle=False) or TimeSeriesSplit.
Groups
If one customer, patient or farm appears in many rows, keep all of their rows on the same side of the split (see GroupShuffleSplit). Otherwise the model can recognise the individual rather than learn the general pattern.
Preprocessing before splitting. Scaling, imputing missing values or selecting features on the whole dataset and then splitting lets test information leak into training. The right order is: split first, then fit every preprocessing step on the training set only. Pipelines make this automatic.
Duplicate rows that end up in both sets. The model gets "test" questions it has literally memorised. Remove duplicates before splitting.
Key points
- Hold back a test set to measure generalisation honestly.
- Use
random_statefor reproducibility andstratify=yfor classification. - Use a validation set or cross-validation for choices; use the test set once.
- Don't shuffle time series; keep groups together; split before preprocessing.
Tip: use the ← and → keys to move between pages.
