Overfitting is a moment when the model:
Perfectly knows train data and Performs poorly on new data
It memorizes, not generalizes.
➡️ How to fight this in practice?
1. More Data
The most reliable way.
If there's not enough data:
☞ collect new data
☞ do data augmentation
☞ use synthetic data
More diversity = less chance of memorizing noise.
2. Regularization
Add a penalty for model complexity.
Main options:
☞ L2 (weight decay)
☞ L1
Less weight → simpler model → less overfitting.
3. Dropout
During training, random neurons are "turned off".
What happens:
☞ the model can't rely on specific connections
☞ learns to be more robust
Usually used:
☞ 0.2 – 0.5
4. Early Stopping
Monitor validation:
☞ train loss drops
☞ val loss first drops, then rises
We stop training when val loss starts rising.
This is one of the most effective methods.
5. Simplify the Model
Sometimes the solution is obvious:
☞ fewer layers
☞ fewer parameters
☞ simpler architecture
A larger model is easier to overfit.
6. Data Augmentation
Especially important for:
CV:
☞ rotations
☞ noise
☞ crops
NLP:
☞ rephrasing
☞ substitutions
The model sees more variants of the same thing.
7. Batch Normalization
Helps:
☞ stabilize training
☞ slightly reduce overfitting
Not the main solution, but it reinforces the others.
8. Proper Validation
If the split is bad, you won't notice the problem.
Use:
☞ train / val / test
☞ k-fold with small data
Otherwise, you'll be optimizing an illusion.
Main Insight: overfitting is a signal:
☞ either not enough data
☞ or model is too complex
☞ or training is set up incorrectly
In One Sentence: To reduce overfitting - add data or reduce model complexity.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore