Imagine you are choosing a hospital for a surgery.
• Hospital A has a higher survival rate than Hospital B for "Easy" cases.
• Hospital A also has a higher survival rate for "Hard" cases.
Common sense says: Choose Hospital A. But when you look at the total combined data, Hospital B actually has a higher survival rate. 🤯
This is Simpson’s Paradox: A trend appears in several different groups but disappears or reverses when these groups are combined.
🔍 Why does this happen?
It happens because of a Lurking Variable (a hidden factor). In this case, Hospital A is a world-class facility, so it takes on way more "Hard" cases than Hospital B. Even though they are better at both types, the high volume of risky surgeries drags their overall average down.
🐍 See the Paradox in Code
Let's simulate this "impossible" scenario using Python:
import pandas as pd
# Data: [Successes, Total Attempts]
data = {
'Hospital': ['A', 'A', 'B', 'B'],
'Case_Type': ['Easy', 'Hard', 'Easy', 'Hard'],
'Survived': [95, 10, 90, 70],
'Total': [100, 100, 100, 1000]
}
df = pd.DataFrame(data)
# 1. Check rates per group
df['Rate'] = df['Survived'] / df['Total']
print("--- Rates by Group ---")
print(df[['Hospital', 'Case_Type', 'Rate']])
# 2. Check overall rates
overall = df.groupby('Hospital').sum()
overall['Overall_Rate'] = overall['Survived'] / overall['Total']
print("\n--- Overall Rates (The Paradox!) ---")
print(overall['Overall_Rate'])
The Result:
• A is better at Easy (95% vs 90%).
• A is better at Hard (10% vs 7%).
• BUT... Overall, B wins (14% vs 52%) because B mostly did "Easy" cases.
🛠 How to avoid being fooled?
1. Don't trust the aggregate: When analyzing data, always try to "segment" or "drill down" into sub-groups.
2. Look for the Weight: Ask yourself: "Is one group disproportionately represented in the total?"
3. Identify the Lurking Variable: What context is missing? (e.g., Age, Severity, Time of Day).
🎯 The Takeaway
In Data Science, the "Big Picture" can sometimes be a big lie. If your analysis produces a result that defies logic, you might be looking at a Simpson’s Paradox. Always slice your data before you trust it.