Skip to content
Kuza Ndoto Academy

K-Nearest Neighbours

IntermediateChapter 3 · Page 12 of 264 min readUpdated 15 Nov 2026

K-Nearest Neighbours (KNN) predicts by looking at the k training examples most similar to a new case and letting them vote. It's one of the most intuitive algorithms in machine learning: "tell me who your neighbours are, and I'll tell you who you are."

How KNN works

  1. Choose a number of neighbours, k (for example, 3).
  2. For a new example, measure its distance to every training example.
  3. Pick the k closest ones.
  4. Classification: predict the most common class among them. Regression: predict the average of their values.

KNN has no real training phase: fit just stores the data. All the work happens at prediction time, which is why it's called a lazy learner.

Measuring distance

The usual measure is Euclidean distance, the straight-line distance you know from Pythagoras. For two points (x₁, y₁) and (x₂, y₂):

d = √((x₁ − x₂)² + (y₁ − y₂)²)

With more features you add one squared difference per feature. Another option is Manhattan distance, the sum of absolute differences, like walking along a grid of streets.

A worked example

A shop describes customers by visits per month and average spend per visit (in thousands of KES). Six known customers are labelled "budget" or "premium". A new customer visits 4 times a month and spends 3 thousand per visit. Which group?

import math

X = [[1, 2], [2, 3], [3, 1], [6, 5], [7, 7], [8, 6]]
y = ["budget", "budget", "budget", "premium", "premium", "premium"]
new = [4, 3]

for point, label in zip(X, y):
    print(point, label, f"{math.dist(new, point):.3f}")
[1, 2] budget 3.162 [2, 3] budget 2.000 [3, 1] budget 2.236 [6, 5] premium 2.828 [7, 7] premium 5.000 [8, 6] premium 5.000

Check one by hand: from (4, 3) to (3, 1) the differences are 1 and 2, so d = √(1 + 4) = √5 ≈ 2.236. The three nearest are [2, 3] (2.000, budget), [3, 1] (2.236, budget) and [6, 5] (2.828, premium). Two budget votes against one premium, so the prediction is budget.

KNN in scikit-learn

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X, y)
print(knn.predict([new]))

distances, indices = knn.kneighbors([new])
print(distances.round(3))
print(indices)
['budget'] [[2. 2.236 2.828]] [[1 2 3]]

kneighbors confirms the hand calculation: the neighbours are the training rows at positions 1, 2 and 3.

02468 02468 visits per month spend per visit (KES '000) 2.0 2.24 2.83 budget premium new customer (4, 3) k = 3 nearest: 2.0 budget, 2.24 budget, 2.83 premium prediction: budget
Figure 1. The dashed circle has radius 2.83, the distance to the third-nearest neighbour. Inside it are two budget customers and one premium customer, so the vote is budget. Both axes use the same scale (34 px per unit), so distances look true to size.

Choosing k

  • Small k (such as 1): very sensitive to noise; one mislabelled neighbour changes the answer. This overfits.
  • Large k: smoother, more stable predictions, but too large and the model ignores local patterns and drifts towards the majority class. This underfits.
  • For two classes, use an odd k to avoid ties.
  • Pick k with cross-validation: try 1, 3, 5, … 21 and keep the best.

Setting weights="distance" gives closer neighbours a bigger vote, which often helps.

Scaling is essential

Because KNN relies entirely on distances, features with big numbers dominate. Suppose customers are described by age and monthly income in KES. Two people aged 20 and 60 earning KES 50,000 and 50,100 are a distance of about √(40² + 100²) ≈ 108 apart, while two people of the same age earning KES 50,000 and 50,200 are 200 apart. The model thinks the age gap barely matters. Always standardise first:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
Common mistake

Running KNN on unscaled features, or fitting the scaler on all the data before splitting. The first lets one feature dominate; the second leaks test information into training. Use a pipeline and fit it on training data only. See Feature Scaling.

Strengths and weaknesses

  • Strengths: simple, no assumptions about the shape of the data, handles many classes naturally, and new data can be added without retraining.
  • Weaknesses: prediction is slow on large datasets (it compares with every stored example), it needs scaling, and it struggles with many features because distances become less meaningful in high dimensions (the "curse of dimensionality"). Reducing features with PCA can help.

Key points

  • KNN predicts from the k most similar training examples: a vote for classes, an average for numbers.
  • Euclidean distance is the square root of the sum of squared feature differences.
  • Choose k with cross-validation; small k overfits, large k underfits.
  • Always scale features before using KNN.
Up next · Supervised LearningRandom ForestsMany trees are better than one.
→

Tip: use the ← and → keys to move between pages.