Evaluation Metrics
An evaluation metric turns "how good is my model?" into a number. Choosing the right metric is as important as choosing the right algorithm, because a model can score brilliantly on one metric while failing at the job you actually care about.
The confusion matrix
For classification, every metric starts from the confusion matrix, which counts the four possible outcomes. Take a fraud detector where "positive" means fraud:
| Predicted: not fraud | Predicted: fraud | |
|---|---|---|
| Actually not fraud | TN (true negative) | FP (false positive, a false alarm) |
| Actually fraud | FN (false negative, a missed fraud) | TP (true positive) |
Here is a model's output for 10 transactions, 4 of them really fraudulent:
from sklearn.metrics import confusion_matrix y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0] # 1 = fraud y_pred = [1, 1, 0, 0, 1, 0, 0, 0, 0, 0] print(confusion_matrix(y_true, y_pred))
scikit-learn puts actual classes in rows and predicted classes in columns, in sorted order (0 then 1). So TN = 5, FP = 1, FN = 2, TP = 2.
Accuracy, precision, recall and F1
- Accuracy = (TP + TN) / total = (2 + 5) / 10 = 0.70. The share of all predictions that were right.
- Precision = TP / (TP + FP) = 2 / 3 ≈ 0.667. Of the transactions we flagged, how many were really fraud? Low precision means many false alarms.
- Recall (sensitivity, true positive rate) = TP / (TP + FN) = 2 / 4 = 0.50. Of all real frauds, how many did we catch? Low recall means many misses.
- F1 score = 2 × precision × recall / (precision + recall) = 2 × 0.667 × 0.5 / 1.167 ≈ 0.571. The harmonic mean, which is only high when both precision and recall are high.
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
print(f"Accuracy: {accuracy_score(y_true, y_pred):.3f}")
print(f"Precision: {precision_score(y_true, y_pred):.3f}")
print(f"Recall: {recall_score(y_true, y_pred):.3f}")
print(f"F1: {f1_score(y_true, y_pred):.3f}")
classification_report(y_true, y_pred) prints all of these for every class in one table.
Which metric matters?
| Situation | Costly mistake | Focus on |
|---|---|---|
| Screening for a serious disease | Missing a sick patient (FN) | Recall |
| Spam filter | Hiding a real email (FP) | Precision |
| Mobile money fraud alerts | Both misses and blocking honest customers | F1, or precision and recall together |
| Balanced classes, equal costs | Any error | Accuracy is fine |
Precision and recall pull against each other. Lowering a classifier's probability threshold flags more cases, which raises recall and usually lowers precision. The ROC curve shows the trade-off at every threshold, and its area (ROC AUC, from roc_auc_score) summarises it: 0.5 is random guessing, 1.0 is perfect ranking. For rare positives, the precision-recall curve is often more informative.
Using accuracy on imbalanced data. If 990 of 1,000 transactions are legitimate, a "model" that always says "not fraud" scores 99% accuracy while catching zero fraud (recall = 0). Whenever one class is rare, report precision, recall, F1 or the confusion matrix, and compare against this kind of trivial baseline.
Regression metrics
For numeric predictions, metrics measure the size of the errors. Take actual values 3, 5, 7 and predictions 2, 5, 9, so the errors are −1, 0 and 2:
- MAE (mean absolute error) = (1 + 0 + 2) / 3 = 1.0. Easy to explain: "on average we're 1 unit off."
- MSE (mean squared error) = (1 + 0 + 4) / 3 ≈ 1.667. Punishes large errors more.
- RMSE = √MSE ≈ 1.291. Back in the original units.
- R² = 1 − (sum of squared errors / sum of squared deviations from the mean) = 1 − 5 / 8 = 0.375. The mean of the actual values is 5, and the squared deviations are 4, 0 and 4.
from sklearn.metrics import (mean_absolute_error, mean_squared_error,
root_mean_squared_error, r2_score)
actual = [3, 5, 7]
predicted = [2, 5, 9]
print(f"MAE: {mean_absolute_error(actual, predicted):.3f}")
print(f"MSE: {mean_squared_error(actual, predicted):.3f}")
print(f"RMSE: {root_mean_squared_error(actual, predicted):.3f}")
print(f"R2: {r2_score(actual, predicted):.3f}")
root_mean_squared_error was added in scikit-learn 1.4. Older tutorials use mean_squared_error(..., squared=False), which is deprecated.
Key points
- The confusion matrix counts TP, TN, FP and FN; most classification metrics come from it.
- Precision asks "how many flagged were right?"; recall asks "how many positives did we catch?"
- Accuracy misleads on imbalanced data; choose metrics that match the cost of each mistake.
- For regression, use MAE, RMSE and R².
Tip: use the ← and → keys to move between pages.
