Skip to content
Kuza Ndoto Academy

Evaluation Metrics

IntermediateChapter 5 · Page 16 of 264 min readUpdated 15 Nov 2026

An evaluation metric turns "how good is my model?" into a number. Choosing the right metric is as important as choosing the right algorithm, because a model can score brilliantly on one metric while failing at the job you actually care about.

The confusion matrix

For classification, every metric starts from the confusion matrix, which counts the four possible outcomes. Take a fraud detector where "positive" means fraud:

Predicted: not fraudPredicted: fraud
Actually not fraudTN (true negative)FP (false positive, a false alarm)
Actually fraudFN (false negative, a missed fraud)TP (true positive)

Here is a model's output for 10 transactions, 4 of them really fraudulent:

from sklearn.metrics import confusion_matrix

y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]   # 1 = fraud
y_pred = [1, 1, 0, 0, 1, 0, 0, 0, 0, 0]

print(confusion_matrix(y_true, y_pred))
[[5 1] [2 2]]

scikit-learn puts actual classes in rows and predicted classes in columns, in sorted order (0 then 1). So TN = 5, FP = 1, FN = 2, TP = 2.

Predicted: not fraud (0) Predicted: fraud (1) Actual: not fraud (0) Actual: fraud (1) 5TN: true negative 1FP: false alarm 2FN: missed fraud 2TP: fraud caught Accuracy 7/10 = 0.70 Precision 2/3 = 0.667 Recall 2/4 = 0.50
Figure 1. The confusion matrix from the code. Correct predictions sit on the navy diagonal; the red cells are the two kinds of mistake.

Accuracy, precision, recall and F1

  • Accuracy = (TP + TN) / total = (2 + 5) / 10 = 0.70. The share of all predictions that were right.
  • Precision = TP / (TP + FP) = 2 / 3 ≈ 0.667. Of the transactions we flagged, how many were really fraud? Low precision means many false alarms.
  • Recall (sensitivity, true positive rate) = TP / (TP + FN) = 2 / 4 = 0.50. Of all real frauds, how many did we catch? Low recall means many misses.
  • F1 score = 2 × precision × recall / (precision + recall) = 2 × 0.667 × 0.5 / 1.167 ≈ 0.571. The harmonic mean, which is only high when both precision and recall are high.
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print(f"Accuracy:  {accuracy_score(y_true, y_pred):.3f}")
print(f"Precision: {precision_score(y_true, y_pred):.3f}")
print(f"Recall:    {recall_score(y_true, y_pred):.3f}")
print(f"F1:        {f1_score(y_true, y_pred):.3f}")
Accuracy: 0.700 Precision: 0.667 Recall: 0.500 F1: 0.571

classification_report(y_true, y_pred) prints all of these for every class in one table.

Which metric matters?

SituationCostly mistakeFocus on
Screening for a serious diseaseMissing a sick patient (FN)Recall
Spam filterHiding a real email (FP)Precision
Mobile money fraud alertsBoth misses and blocking honest customersF1, or precision and recall together
Balanced classes, equal costsAny errorAccuracy is fine

Precision and recall pull against each other. Lowering a classifier's probability threshold flags more cases, which raises recall and usually lowers precision. The ROC curve shows the trade-off at every threshold, and its area (ROC AUC, from roc_auc_score) summarises it: 0.5 is random guessing, 1.0 is perfect ranking. For rare positives, the precision-recall curve is often more informative.

Common mistake

Using accuracy on imbalanced data. If 990 of 1,000 transactions are legitimate, a "model" that always says "not fraud" scores 99% accuracy while catching zero fraud (recall = 0). Whenever one class is rare, report precision, recall, F1 or the confusion matrix, and compare against this kind of trivial baseline.

Regression metrics

For numeric predictions, metrics measure the size of the errors. Take actual values 3, 5, 7 and predictions 2, 5, 9, so the errors are −1, 0 and 2:

  • MAE (mean absolute error) = (1 + 0 + 2) / 3 = 1.0. Easy to explain: "on average we're 1 unit off."
  • MSE (mean squared error) = (1 + 0 + 4) / 3 ≈ 1.667. Punishes large errors more.
  • RMSE = √MSE ≈ 1.291. Back in the original units.
  • R² = 1 − (sum of squared errors / sum of squared deviations from the mean) = 1 − 5 / 8 = 0.375. The mean of the actual values is 5, and the squared deviations are 4, 0 and 4.
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
                             root_mean_squared_error, r2_score)

actual = [3, 5, 7]
predicted = [2, 5, 9]
print(f"MAE:  {mean_absolute_error(actual, predicted):.3f}")
print(f"MSE:  {mean_squared_error(actual, predicted):.3f}")
print(f"RMSE: {root_mean_squared_error(actual, predicted):.3f}")
print(f"R2:   {r2_score(actual, predicted):.3f}")
MAE: 1.000 MSE: 1.667 RMSE: 1.291 R2: 0.375
Note

root_mean_squared_error was added in scikit-learn 1.4. Older tutorials use mean_squared_error(..., squared=False), which is deprecated.

Key points

  • The confusion matrix counts TP, TN, FP and FN; most classification metrics come from it.
  • Precision asks "how many flagged were right?"; recall asks "how many positives did we catch?"
  • Accuracy misleads on imbalanced data; choose metrics that match the cost of each mistake.
  • For regression, use MAE, RMSE and R².
Up next · Model EvaluationOverfitting & UnderfittingWhen models learn too much or too little.
→

Tip: use the ← and → keys to move between pages.