What is Overfitting?
Overfitting happens when your model learns the training data too well, including noise and minor patterns.
Result: Performs well on training data, poorly on new/unseen data.
Signs of Overfitting:
⦁ High training accuracy
⦁ Low testing accuracy
⦁ Large gap between training and test performance
Why It Happens:
⦁ Too complex models (e.g., deep trees, too many layers)
⦁ Small training dataset
⦁ Too many features
⦁ Training for too many epochs
Visual Example:
⦁ Underfitting: Straight line → misses pattern
⦁ Good Fit: Smooth curve → generalizes well
⦁ Overfitting: Zigzag line → memorizes noise
How to Reduce Overfitting (Regularization Techniques):
1️⃣ Simplify the Model
Use fewer features or shallower trees/layers.
2️⃣ Regularization (L1 & L2)
⦁ L1 (Lasso): Can remove unimportant features
⦁ L2 (Ridge): Penalizes large weights, keeps all features
Both add penalty terms to the loss function.
3️⃣ Cross-Validation
Helps detect and prevent overfitting by validating on multiple data splits.
4️⃣ Pruning (for Decision Trees)
Remove branches that don’t improve performance on test data.
5️⃣ Early Stopping (in Neural Nets)
Stop training when validation error starts increasing.
6️⃣ Dropout (for Deep Learning)
Randomly ignore neurons during training to prevent dependency.
Python Example (L2 Regularization with Logistic Regression):
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(penalty='l2', C=0.1)
model.fit(X_train, y_train)
Summary:
⦁ Overfitting = Memorizing training data
⦁ Regularization = Force model to stay general
⦁ Goal = Balance bias and variance
💬 Tap ❤️ for more