Your model to detect a rare disease (1% prevalence) boasts 99% accuracy. Impressive? Not if it just says "NO DISEASE" to everyone! For imbalanced data, plain accuracy is a lie.
📈 The Problem: Imbalanced Data
Many real-world cases (fraud, disease, ad clicks) have a tiny "positive" class. A model predicting the majority class (e.g., "no disease") will have high accuracy but be useless for finding the rare events you care about.
📊 Beyond Accuracy: The Confusion Matrix
Break down predictions into:
• True Positives (TP): Correctly found the positive.
• True Negatives (TN): Correctly found the negative.
• False Positives (FP): Wrongly said positive (costly "false alarms").
• False Negatives (FN): Wrongly said negative (costly "missed opportunities").
🎯 The Right Metrics
• Accuracy:
(TP+TN) / Total - Avoid for imbalanced data!• Precision:
TP / (TP + FP)• Meaning: Out of all times it said "Positive," how many were truly positive?
• Use When: False Positives (FP) are very costly (e.g., wrongly flagging a healthy person as sick).
• Recall:
TP / (TP + FN)• Meaning: Out of all actual positives, how many did it catch?
• Use When: False Negatives (FN) are very costly (e.g., missing a real fraud, not detecting a tumor).
• F1-Score: Balances Precision and Recall.
🐍 Code Example: The 99% Accurate Lie
from sklearn.metrics import accuracy_score, precision_score, recall_score
import numpy as np
y_true = np.concatenate([np.zeros(990), np.ones(10)]) # 1000 samples, 1% positive
# Model 1: Always predicts '0' (no disease)
y_pred_bad = np.zeros(1000)
print(f"Model 1 (Always No Disease):\n Accuracy: {accuracy_score(y_true, y_pred_bad):.2f}")
print(f" Precision: {precision_score(y_true, y_pred_bad, zero_division=0):.2f}") # 0.00!
print(f" Recall: {recall_score(y_true, y_pred_bad):.2f}\n") # 0.00!
# Model 2: Catches 5 positives, 2 false alarms (Better!)
y_pred_better = np.zeros(1000)
y_pred_better[990:995] = 1 # 5 True Positives
y_pred_better[100:102] = 1 # 2 False Positives
print(f"Model 2 (Actually Catches Some):\n Accuracy: {accuracy_score(y_true, y_pred_better):.2f}")
print(f" Precision: {precision_score(y_true, y_pred_better, zero_division=0):.2f}") # 0.71
print(f" Recall: {recall_score(y_true, y_pred_better):.2f}") # 0.50
# Model 2's accuracy might be slightly lower, but its Precision/Recall shows it's far superior!
🎯 Today's Goal (What you should do)
✔️ Recognize accuracy's flaw for imbalanced data.
✔️ Pick Precision when False Positives hurt most.
✔️ Pick Recall when False Negatives hurt most.
✔️ Understand what your model's mistakes truly cost.