Here's where conformal prediction comes to rescue. This approach adds an uncertainty estimate to model predictions.
Regular model says demand tomorrow will be 100 units and Conformal prediction says, demand tomorrow will likely be between 80 and 125 units.
In classification: a regular model says image shows object A and Conformal prediction says the most likely object in the image is A or C.
If we formalize problem, we want to construct such intervals or sets of answers that will contain correct answer in approximately p% of cases on new data. Size of this set shows the model's uncertainty: the wider it is, the less confident the model appears to be.
➡️ Where, How, and What tools to use?
Where can this be useful?
• Medicine. In medical tasks, one confident but incorrect diagnosis can be costly. Conformal prediction allows the model not to pretend to be omniscient. Instead of one diagnosis, it can give several likely options. For the doctor, this is an additional hint where to look more closely.
• Predictive maintenance. Instead of a point estimate, we get a risk window when important equipment fails.
• Retail. A forecast like "we'll sell 407 packages of ice cream tomorrow" looks too magical and doesn't give a full picture of demand. A range of forecasts is more convenient for inventory management: we can estimate the risk of a shortage or an overstocked warehouse.
How does it work?
1. We have a trained model. We take a separate part of the data that the model did not see during training. This is usually called a calibration sample.
2. On this sample, we look at how much the model is wrong.
3. We collect such errors and choose a threshold that covers the necessary percentage of cases. For example, we want 90% reliability - we take such a value of error that approximately 90% of past correct answers fall within the corresponding interval.
4. After this, for a new object, the model makes a prediction, and we add an interval around it.
The main idea: if the model on similar data usually made errors no more than a certain value, we use this value as a protective "gap" around future predictions.
It's important to remember that conformal prediction works well when the future data are similar to those on which we calibrated. But if the world has changed dramatically - for example, there was a pandemic, a crisis, or a sudden hype on social networks - past errors may not describe the future well enough.
How to USE it?
For a quick start, you can take a look at MAPIE. This is a library in the style of scikit-learn that allows you to build prediction intervals for regression and prediction sets for classification. Another good option is PUNCC. It's more flexible and allows you to apply conformal prediction on top of various models, including sklearn models, neural networks, and custom pipelines.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore