Skip to content
Kuza Ndoto Academy

Cross-Validation

IntermediateChapter 5 · Page 18 of 264 min readUpdated 15 Nov 2026

A single train/test split gives one score that depends on which rows happened to land in the test set. Cross-validation repeats the split several times and averages the results, giving a more reliable estimate of how well a model will perform, and a fair way to compare models and settings.

K-fold cross-validation

  1. Split the training data into k equal parts called folds (5 or 10 is common).
  2. Train on k − 1 folds and test on the remaining fold.
  3. Repeat k times, so each fold is the test fold exactly once.
  4. Report the mean (and spread) of the k scores.

Every row is used for testing once and for training k − 1 times. With 5 folds and 100 rows, each model trains on 80 rows and tests on 20.

import numpy as np
from sklearn.model_selection import KFold

X = np.arange(10).reshape(10, 1)     # 10 rows
kf = KFold(n_splits=5)

for fold, (train_idx, test_idx) in enumerate(kf.split(X), start=1):
    print(fold, test_idx, len(train_idx))
1 [0 1] 8 2 [2 3] 8 3 [4 5] 8 4 [6 7] 8 5 [8 9] 8
rows 0-1rows 2-3rows 4-5rows 6-7rows 8-9 Fold 1 Test Train Train Train Train Fold 2 Train Test Train Train Train Fold 3 Train Train Test Train Train Fold 4 Train Train Train Test Train Fold 5 Train Train Train Train Test final score = mean of the 5 test-fold scores
Figure 1. The five folds from the KFold output above. Every row is tested exactly once and used for training four times.

Without shuffling, folds are consecutive blocks. If your data is sorted (for example, all class 0 first), use KFold(n_splits=5, shuffle=True, random_state=42).

cross_val_score: the easy way

from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

X, y = load_iris(return_X_y=True)
model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))

scores = cross_val_score(model, X, y, cv=5)
print(scores.round(3))
print(f"Mean accuracy: {scores.mean():.3f} (std {scores.std():.3f})")

You'll see five accuracy scores, one per fold, mostly between about 0.93 and 1.00, with a mean of around 0.95 to 0.97. The standard deviation tells you how stable the model is: a large spread means the score depends heavily on which rows are tested.

Note

When you pass cv=5 with a classifier, scikit-learn automatically uses StratifiedKFold, which keeps the class proportions the same in every fold. For regression it uses plain KFold.

Tuning settings with GridSearchCV

Settings you choose before training, like k in KNN or max_depth in a tree, are called hyperparameters. GridSearchCV tries every combination you list, scores each one with cross-validation, and keeps the best:

from sklearn.model_selection import GridSearchCV, train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

param_grid = {"kneighborsclassifier__n_neighbors": [1, 3, 5, 7, 9, 11]}
search = GridSearchCV(model, param_grid, cv=5)
search.fit(X_train, y_train)

print(search.best_params_)
print(f"Best CV accuracy: {search.best_score_:.3f}")
print(f"Test accuracy: {search.score(X_test, y_test):.3f}")

The best k will be one of the listed values (often in the middle of the range for iris), and both accuracies should be well above 0.9. Note the parameter name: in a pipeline built with make_pipeline, each step is named after its class in lowercase, followed by two underscores and the parameter. After the search, search is refitted on all the training data with the best setting, so you can use it directly. For large grids, RandomizedSearchCV tries a random sample of combinations instead.

Other splitting strategies

SplitterWhen to use it
StratifiedKFoldClassification, especially with imbalanced classes
TimeSeriesSplitData ordered in time: always trains on the past and tests on the future
GroupKFoldSeveral rows per customer, patient or school: keeps each group in one fold
LeaveOneOutVery small datasets (one row per test fold; slow for large data)
Common mistake

Preprocessing outside cross-validation. If you scale or impute the whole dataset first and then run cross_val_score on the result, each test fold has already influenced the preprocessing. Put every step in a Pipeline and pass the pipeline to cross_val_score, so the steps are refitted on the training folds only, every time.

Common mistake

Reporting best_score_ from a grid search as your final result. That score was used to choose the settings, so it's slightly optimistic. Report the score on a separate test set (or use nested cross-validation).

Key points

  • K-fold cross-validation trains and tests k times and averages the scores.
  • It gives a more reliable estimate than a single split and shows how stable a model is.
  • GridSearchCV uses cross-validation to choose hyperparameters.
  • Keep preprocessing inside a pipeline and keep a final test set untouched.
Up next · New chapter: Deep Learning & GenAINeural NetworksLayers of artificial neurons.
→

Tip: use the ← and → keys to move between pages.