Skip to content
Kuza Ndoto Academy

Bias & Fairness

IntermediateChapter 7 · Page 24 of 264 min readUpdated 15 Nov 2026

A machine learning model can treat groups of people unfairly even when nobody intended it to. Because models learn from historical data, they can copy and even amplify past discrimination. Understanding where bias comes from, and how to measure it, is part of building any model that affects people's lives.

Where bias comes from

SourceWhat happensExample
Historical biasThe data accurately reflects an unfair pastA hiring model trained on years of decisions that favoured men learns to favour men
Representation (sampling) biasSome groups are missing or rare in the dataA speech recogniser trained mostly on American English struggles with Kenyan accents or Sheng
Measurement biasA feature or label measures different groups differentlyUsing "number of past arrests" as a stand-in for crime, when some areas are policed far more heavily
Label biasThe "correct answers" contain human prejudicePast loan officers' subjective decisions used as labels
Proxy featuresA neutral-looking feature stands in for a protected attributeHome area or surname correlating with ethnicity
Deployment and feedback loopsThe model is used on a different population, or its decisions shape future dataA credit model built on smartphone users applied to people with basic phones
Common mistake

Thinking that deleting the gender or ethnicity column makes a model fair. Other features, such as location, name, school or shopping patterns, can act as proxies and recreate the same bias. You often need the sensitive attribute in your evaluation data precisely so you can check for unfairness.

Real-world examples

  • Hiring: in 2018 Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it downgraded CVs that included the word "women's", as in "women's chess club". It had learned from a decade of mostly male applicants.
  • Face analysis: the 2018 Gender Shades study by Joy Buolamwini and Timnit Gebru found that commercial gender-classification systems had error rates of up to about 35% for darker-skinned women, compared with under 1% for lighter-skinned men.
  • Criminal justice: a 2016 ProPublica investigation of the COMPAS risk tool in the USA found that Black defendants who did not reoffend were much more likely than white defendants to have been labelled high risk. The tool's developer disputed the analysis, and the debate showed that different fairness definitions can conflict.

Measuring fairness

There are many fairness metrics. Three common ones compare groups:

  • Demographic parity: each group should receive positive predictions (such as loan approvals) at a similar rate, called the selection rate.
  • Equal opportunity: among people who truly deserve a positive outcome, each group should have a similar true positive rate (recall).
  • Equalised odds: both the true positive rate and the false positive rate should be similar across groups.

A simple summary is the disparate impact ratio: the lower group's selection rate divided by the higher group's. A widely quoted rule of thumb from US employment guidance, the "four-fifths rule", treats a ratio below 0.8 as a warning sign.

import pandas as pd

df = pd.DataFrame({
    "group":     ["A"] * 5 + ["B"] * 5,
    "actual":    [1, 1, 0, 1, 0,  1, 0, 1, 0, 0],   # 1 = would repay
    "predicted": [1, 1, 1, 0, 0,  0, 0, 1, 0, 0],   # 1 = approved
})

selection = df.groupby("group")["predicted"].mean()
print(selection)
group A 0.6 B 0.2 Name: predicted, dtype: float64
tpr = df[df["actual"] == 1].groupby("group")["predicted"].mean()
print(tpr)
print(f"Disparate impact ratio: {selection['B'] / selection['A']:.2f}")
group A 0.666667 B 0.500000 Name: predicted, dtype: float64 Disparate impact ratio: 0.33
00.250.50.751 0.60.2 0.6670.5 0.8 × 0.6 = 0.48 Selection rate (approved) True positive rate Group A Group B
Figure 1. The rates computed above (scale: 180 px = 1.0). Group B's selection rate of 0.2 falls well below the four-fifths line at 0.48, which is 0.8 times group A's rate.

Group A is approved 3 times out of 5; group B only once. Among people who would actually repay, group A is approved 2 out of 3 times and group B 1 out of 2. The ratio 0.2 / 0.6 ≈ 0.33 is far below 0.8. With only ten rows this is just an illustration, but on real data a result like this demands investigation. The open-source Fairlearn library computes these metrics for you.

Note

Researchers have shown that, when groups have different underlying rates of the outcome, it's mathematically impossible to satisfy several fairness definitions at once. Choosing a metric is a value judgement about which errors matter most, and it should involve the people affected, not just the data team.

Reducing bias

  1. Before training: collect more representative data, check label quality, audit features for proxies, and document the dataset.
  2. During training: reweight under-represented groups or use fairness-aware training methods.
  3. After training: evaluate every metric per group, not just overall; adjust decision thresholds if justified; keep a human review for borderline or high-stakes cases.
  4. After deployment: monitor outcomes by group, provide a way for people to appeal, and retrain when problems appear.
Common mistake

Reporting only overall accuracy. A model can be 95% accurate overall and much worse for a minority group. Always break your evaluation metrics down by group.

Key points

  • Bias enters through history, sampling, measurement, labels, proxies and deployment.
  • Removing a sensitive column doesn't remove bias.
  • Measure fairness with selection rates, true positive rates and ratios across groups.
  • Fairness definitions can conflict; choosing one is a human decision.
Up next · Responsible AIAI Ethics & SafetyUsing AI responsibly.
→

Tip: use the ← and → keys to move between pages.