Bias & Fairness
A machine learning model can treat groups of people unfairly even when nobody intended it to. Because models learn from historical data, they can copy and even amplify past discrimination. Understanding where bias comes from, and how to measure it, is part of building any model that affects people's lives.
Where bias comes from
| Source | What happens | Example |
|---|---|---|
| Historical bias | The data accurately reflects an unfair past | A hiring model trained on years of decisions that favoured men learns to favour men |
| Representation (sampling) bias | Some groups are missing or rare in the data | A speech recogniser trained mostly on American English struggles with Kenyan accents or Sheng |
| Measurement bias | A feature or label measures different groups differently | Using "number of past arrests" as a stand-in for crime, when some areas are policed far more heavily |
| Label bias | The "correct answers" contain human prejudice | Past loan officers' subjective decisions used as labels |
| Proxy features | A neutral-looking feature stands in for a protected attribute | Home area or surname correlating with ethnicity |
| Deployment and feedback loops | The model is used on a different population, or its decisions shape future data | A credit model built on smartphone users applied to people with basic phones |
Thinking that deleting the gender or ethnicity column makes a model fair. Other features, such as location, name, school or shopping patterns, can act as proxies and recreate the same bias. You often need the sensitive attribute in your evaluation data precisely so you can check for unfairness.
Real-world examples
- Hiring: in 2018 Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it downgraded CVs that included the word "women's", as in "women's chess club". It had learned from a decade of mostly male applicants.
- Face analysis: the 2018 Gender Shades study by Joy Buolamwini and Timnit Gebru found that commercial gender-classification systems had error rates of up to about 35% for darker-skinned women, compared with under 1% for lighter-skinned men.
- Criminal justice: a 2016 ProPublica investigation of the COMPAS risk tool in the USA found that Black defendants who did not reoffend were much more likely than white defendants to have been labelled high risk. The tool's developer disputed the analysis, and the debate showed that different fairness definitions can conflict.
Measuring fairness
There are many fairness metrics. Three common ones compare groups:
- Demographic parity: each group should receive positive predictions (such as loan approvals) at a similar rate, called the selection rate.
- Equal opportunity: among people who truly deserve a positive outcome, each group should have a similar true positive rate (recall).
- Equalised odds: both the true positive rate and the false positive rate should be similar across groups.
A simple summary is the disparate impact ratio: the lower group's selection rate divided by the higher group's. A widely quoted rule of thumb from US employment guidance, the "four-fifths rule", treats a ratio below 0.8 as a warning sign.
import pandas as pd
df = pd.DataFrame({
"group": ["A"] * 5 + ["B"] * 5,
"actual": [1, 1, 0, 1, 0, 1, 0, 1, 0, 0], # 1 = would repay
"predicted": [1, 1, 1, 0, 0, 0, 0, 1, 0, 0], # 1 = approved
})
selection = df.groupby("group")["predicted"].mean()
print(selection)
tpr = df[df["actual"] == 1].groupby("group")["predicted"].mean()
print(tpr)
print(f"Disparate impact ratio: {selection['B'] / selection['A']:.2f}")
Group A is approved 3 times out of 5; group B only once. Among people who would actually repay, group A is approved 2 out of 3 times and group B 1 out of 2. The ratio 0.2 / 0.6 ≈ 0.33 is far below 0.8. With only ten rows this is just an illustration, but on real data a result like this demands investigation. The open-source Fairlearn library computes these metrics for you.
Researchers have shown that, when groups have different underlying rates of the outcome, it's mathematically impossible to satisfy several fairness definitions at once. Choosing a metric is a value judgement about which errors matter most, and it should involve the people affected, not just the data team.
Reducing bias
- Before training: collect more representative data, check label quality, audit features for proxies, and document the dataset.
- During training: reweight under-represented groups or use fairness-aware training methods.
- After training: evaluate every metric per group, not just overall; adjust decision thresholds if justified; keep a human review for borderline or high-stakes cases.
- After deployment: monitor outcomes by group, provide a way for people to appeal, and retrain when problems appear.
Reporting only overall accuracy. A model can be 95% accurate overall and much worse for a minority group. Always break your evaluation metrics down by group.
Key points
- Bias enters through history, sampling, measurement, labels, proxies and deployment.
- Removing a sensitive column doesn't remove bias.
- Measure fairness with selection rates, true positive rates and ratios across groups.
- Fairness definitions can conflict; choosing one is a human decision.
Tip: use the ← and → keys to move between pages.
