Almost everyone in ML experiences this problems.
PROBLEM 1:
You cleaned the data, trained the model, spent a lot of time, looked at the metrics and the quality turned out to be much worse than you expected.
The first thought is usually "We need a more complex model." In my experience this is a mistake in 80% of cases. Problem is often not the model at all.
First thing to check is the data. Very often it turns out that the target is noisy, the classes are poorly separated, half of the features are useless, there's not much signal in the data. Some tasks are just hard to predict and that's normal.
There's a feeling that many people expect magic from ML "If the model is smart, it will find everything by itself." It won't, if there's no consistent pattern in the data, XGBoost won't create one.
PROBLEM 2
The second problem is leakage or a bad split especially in tabular data. Sometimes offline everything is beautiful ROC-AUC = 0.95, almost perfect accuracy and then the model falls apart on new data and vice versa.
The metrics are low because the split is too strict and realistic. Another common story is the wrong metric. For example optimizing accuracy with severe imbalance, looking at ROC-AUC where precision matters, rejoicing over a good loss that means nothing to the business
The model can be “mathematically good” and useless at the same time. Baseline is almost always underestimated.
Sometimes logistic regression, the group average, a simple rule by hand give a result close to a complex model and this is not a failure. On the contrary, this is a good sign that the task is either almost linear, or there's not enough data.
There's another unpleasant thing, Some tasks just aren't worth ML. Seriously, it happens that there's not enough data, supporting the model is more expensive than the benefits, the business effect is minimal but many continue tuning the learning rate, changing architectures, running AutoML, going through 40 models because "we're doing AI".
Even though they haven't even looked at the distributions, the model's errors, the quality of the target and that's usually where the answer lies.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore