Skip to content
Kuza Ndoto Academy

Cleaning Data for ML

IntermediateChapter 2 · Page 6 of 264 min readUpdated 15 Nov 2026

Real-world data is messy: blank cells, typing errors, repeated rows and impossible values. Most algorithms either crash on missing values or quietly learn the wrong pattern from bad ones, so cleaning is a required step before any model is trained.

A messy dataset

Here is a small customer table with typical problems. Can you spot them before reading on?

import pandas as pd

df = pd.DataFrame({
    "age": [22, 25, None, 31, 25, 120],
    "town": ["Nairobi", "nairobi ", "Kisumu", "Mombasa", "nairobi ", "Kisumu"],
    "spend": [1200, 800, 950, None, 800, 1100],
})
print(df.isna().sum())
age 1 town 0 spend 1 dtype: int64

Problems: one missing age, one missing spend, "nairobi " written in lowercase with a trailing space, rows 1 and 4 that will be identical once the town is fixed, and an age of 120 that is almost certainly a typing error.

Missing values

You have three main options:

  • Drop rows with df.dropna(). Fine when very few rows are affected, but you lose data, and if the missing rows are different from the rest (for example, poorer customers skipping an income question) you introduce bias.
  • Drop a column when most of its values are missing.
  • Impute (fill in) a value: the median for numbers (robust to outliers), the most frequent value for categories, or a value predicted from other columns.
Tip

The fact that a value is missing can itself be informative. Adding a 0/1 column such as income_missing before filling lets the model learn from it. scikit-learn's SimpleImputer(add_indicator=True) does this for you.

Inconsistent text and duplicates

"Nairobi", "nairobi " and "NAIROBI" are the same town to you but three different categories to a model. Standardise text first, then remove duplicates, because some duplicates only appear after the text is fixed.

Outliers

An outlier is a value far from the rest. Some are errors (age 120, a negative price); some are real but rare (a customer who spends KES 2 million). Fix or remove errors; think carefully before removing real values, since they may be exactly what you care about, as in fraud detection.

A common rule flags values outside Q1 − 1.5 × IQR and Q3 + 1.5 × IQR, where IQR = Q3 − Q1. For the ages 22, 25, 25, 31 and 120: Q1 = 25, Q3 = 31, IQR = 6, so the upper fence is 31 + 9 = 40. The value 120 is far above it.

Putting it together

df["town"] = df["town"].str.strip().str.title()   # "nairobi " becomes "Nairobi"
df = df.drop_duplicates()                          # row 4 now duplicates row 1

df.loc[df["age"] > 100, "age"] = None              # impossible age becomes missing
df["age"] = df["age"].fillna(df["age"].median())
df["spend"] = df["spend"].fillna(df["spend"].median())
print(df)
age town spend 0 22.0 Nairobi 1200.0 1 25.0 Nairobi 800.0 2 25.0 Kisumu 950.0 3 31.0 Mombasa 1025.0 5 25.0 Kisumu 1100.0

Check the numbers: after the duplicate is dropped, the known ages are 22, 25 and 31, so the median is 25. The known spends are 800, 950, 1100 and 1200, so the median is (950 + 1100) / 2 = 1025. Notice that index 4 is gone; call df.reset_index(drop=True) if you want 0 to 4 again. For more pandas techniques see Cleaning Data.

Cleaning without leaking

The median above was calculated on the whole dataset. That's fine for exploring, but in a real modelling project any value you learn from the data (a median, a mean, a list of categories) must come from the training set only. The safest way is to put imputation inside a pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression

model = make_pipeline(SimpleImputer(strategy="median"), LinearRegression())
# model.fit(X_train, y_train) learns the medians from X_train only,
# then reuses those same medians when you call model.predict(X_test)
Common mistake

Data leakage through cleaning: filling missing values, removing outliers or scaling using statistics from the full dataset, then splitting into train and test. The test set has influenced training, so your scores look better than they will be on truly new data. Split first (see Train/Test Split), then fit every cleaning step on the training data.

Key points

  • Count missing values with isna().sum(), then drop or impute them deliberately.
  • Standardise text before removing duplicates.
  • Fix impossible values; think hard before deleting genuine rare ones.
  • Learn cleaning statistics from training data only, ideally inside a pipeline.
Up next · Data PreparationFeature ScalingNormalisation and standardisation.
→

Tip: use the ← and → keys to move between pages.