In the previous post, i explained what conformal prediction is and why it's needed. But you probably have some questions:

1. How does the model understand how "strange" or "risky" a particular object is?
2. How is the conformal predictor trained?

The answer to the first question: through the measure of discomfort.

The measure of discomfort is a function that shows how poorly a particular pair (x, y) matches the model and the already known data.

The measures of discomfort can be simple functions, such as MAE or hinge_loss:

nonconformity_mae = |y_true - y_pred|
nonconformity_hinge = 1 - P(true_class)
or more complex ones, such as Brier's score.

An intuitive example

Suppose the model classifies images. For a normal picture, the model says:

Barbie: 0.02
Ken: 0.97
Oppenheimer: 0.01

And for a blurry picture of an animal in the forest:

Barbie: 0.2
Ken: 0.42
Oppenheimer: 0.38

A regular classifier in all cases will choose the Ken class. Conformal prediction in the second case may say:

{Ken, Oppenheimer}

Because the measure of discomfort for these classes will not be high enough to reject them outright.

Next, let's talk about how to train it.

TCP: Transductive Conformal Prediction

TCP, strictly speaking, is not "trained" like a regular model. It's better to formulate it this way:

In TCP, for each new object and each possible answer, we temporarily add this answer to the training set, retrain or re-evaluate the model, and check how "uncomfortable" this answer is relative to the rest of the data.

Let's consider the TCP algorithm step by step.

Suppose there is a sample:

D = {(x1, y1), ..., (xn, yn)}

and a new object

x_new.

• Step 1. Take one of the classes, for example, Barbie. Make an assumption:

y_new = Barbie

• Step 2. Add it to the existing dataset:

D_Barbie = D ∪ {(x_new, Barbie)}

• Step 3. Train the model on the new set.
• Step 4. Calculate the measure of discomfort, for example, hinge_loss, for all objects, including the new one.
• Step 5. Compare the new object with the rest. See how the discomfort of the new object is relative to the rest of the objects in D_Barbie. Simplified:

p_value(Barbie) = the proportion of objects with a score ≥ score_x_new

Steps 1 to 5 are repeated for each class. The final prediction set is formed from the classes for which p_value > α, where α is the desired significance level.

ICP: Inductive Conformal Prediction

Experienced ML engineers, having read the previous part, are probably horrified. For predicting on 1000 objects in 10 classes, we will need 10,000 retrains of the model!

This problem is solved by the ICP method at the expense of allocating a separate calibration set:

• Step 1. Divide the data into train and calibration:

D_train  
D_calibration

• Step 2. Train the model on D_train. After this, the model is no longer retrained for each new object.
• Step 3. Calculate the measures of discomfort on D_calibration. For each object from D_calibration, calculate how poorly the model predicted the correct answer. We get a set of calibration scores:

scores = [α1, α2, ..., αm]

• Step 4. Set the significance level α and select the threshold q. Now we select such a threshold q of the calibration scores that the required proportion of calibration scores is not greater than it. Simplified:

q = the 90th percentile of the calibration scores

• Step 5. Apply to the new object. For the new object, calculate the score for each possible class:

score(Barbie) = 1 - P(Barbie)
score(Ken) = 1 - P(Ken)
score(Oppenheimer) = 1 - P(Oppenheimer)

The prediction set will include those classes for which score ≤ q.

The main difference from TCP:

ICP once trains the model and once calibrates the threshold. After that, for new objects, it uses the ready-made model and the ready-made calibration, so it works much faster.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore