Most unpopular fact in ML:
the model isn't the most important thing.
You can spend hours choosing between:
XGBoost, LightGBM, CatBoost ...and get a +1% increase in quality.
But you can change the features - and get a +20% increase.
➡️ Let's figure out why?
The model only learns from what you give it
Garbage in → garbage out
If the features:
- are noisy
- are irrelevant
- don't reflect the task well
👉 no model will save you
Even the biggest one.
Real-life example
Task: predict customer churn
Features:
- age
- city
- tariff
Model: ok, but weak result
Added:
- time since last action
- frequency of use
- change in activity
👉 sharp increase in quality
Why?
Because the features started to reflect real behavior
Feature Engineering = implementing knowledge about the task
The model doesn't know:
- the business
- the context
- the causal relationships
But you do.
And when you create features -
you "embed" this knowledge into the data.
Model vs Features
What we change → effect
Model → +1–5%
Hyperparameters → +1–3%
Feature Engineering → +10–50%
Where FE is especially crucial
- Tabular data
- Small datasets
- Business tasks
👉 where there aren't millions of examples, features are everything
When the model is more important
- CV (images)
- NLP (texts)
- Speech
👉 where features learn automatically
Why everyone ignores FE
Because:
- it's hard
- it takes a long time
- there's no "magic button"
- it requires understanding the data
It's much easier to:
"let's try another model"
Main insight
ML isn't a competition of models.
It's a competition of data representations.
In one sentence: best way to improve a model is to
stop tuning the model and start tuning the data
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore