Conformal intervals in production ML with covariate shift: How to maintain coverage without unnecessarily wide predictions?

Split conformal works well with exchangeability: train, calibration, and test come from same distribution. In production, this often breaks down due to geo, devices, channels, seasonality or a change in acquisition mix and a common mistake is to simply expand intervals "with a margin".

What exactly breaks down?

With covariate shift, we have: p_prod(x) != p_cal(x)
but we assume that p(y|x) approximately holds. If we calculate usual conformal quantile on old calibration set, coverage on current traffic might drop.

Naive solution is to globally increase correction. Coverage will partially recover, but price prediction interval, ETA interval or forecast band will become so wide that downstream system will no longer trust them.

Basic production recipe

1️⃣ Train a quantile model:
q_low(x), q_high(x)

2️⃣ Calculate nonconformity scores on calibration set:
s_i = max(q_low(x_i)-y_i, y_i-q_high(x_i), 0)

3️⃣ Estimate importance weights:
w_i ~= p_prod(x_i) / p_cal(x_i)

4️⃣ Use weighted quantile scores instead of usual ones.

5️⃣ For a new object, construct:
C(x) = [q_low(x)-tau, q_high(x)+tau]

Minimal skeleton:

import numpy as np

def weighted_quantile(values, weights, q):
    order = np.argsort(values)
    v = np.asarray(values)[order]
    w = np.asarray(weights)[order]
    cw = np.cumsum(w)
    return v[np.searchsorted(cw, q * cw[-1])]

alpha = 0.1

scores = np.maximum(q_low_cal - y_cal,
                    y_cal - q_high_cal,
                    0)

weights = ratio_model.predict_weight(X_cal)
tau = weighted_quantile(scores, weights, 1 - alpha)

low = q_low_prod - tau
high = q_high_prod + tau

This way, calibration distribution becomes closer to production distribution without unnecessarily widening all intervals.

How not to get too wide intervals?

One global tau often overestimates uncertainty if model error strongly depends on x.

Practically helps:

- CQR instead of point prediction: Conformalized Quantile Regression already models heteroscedastic uncertainty, so conformal correction is usually smaller.
- Normalized score: for example s_i = |y_i - y_hat_i| / sigma_hat(x_i), and the interval is constructed as y_hat(x) +- tau * sigma_hat(x).
- Local calibration: a separate tau per geo, device, channel, price bucket, or risk bucket. This is close to Mondrian conformal, but requires a sufficient number of calibration examples in each segment.
- Rolling calibration buffer: for recommendations, scoring and forecasting, old calibration set quickly stops describing current traffic mix.

Main risk - bad weights

Density ratio model can be noisy. A few objects with huge weights effectively "replace" entire calibration set.

Control:

ESS = (sum w)^2 / sum(w^2)

If ESS is low, the weighted quantile is unstable and intervals start jumping from release to release.

Practical measures:

- clip weights and monitor proportion of clipped weights;
- smooth the density ratio;
- merge rare segments;
- not calibrate a segment where there are few fresh labels;
- run recalibration when ESS drops or distribution drifts on X.

Production checklist

- a separate calibration set, not mixed with training;
- drift detection on feature distribution;
- density ratio model between prod traffic and calibration traffic;
- weighted conformal calibration;
- monitor coverage, average width, coverage by slices, ESS, and latency;
- alerts on increasing interval width without increasing error;
- A/B validation if intervals affect routing, fallback or human review.

It's important not to confuse marginal and conditional coverage. Conformal can maintain 90% coverage on stream on average, but fail in individual microsegments. This needs to be explicitly checked in production.

With covariate shift, goal is not to blindly widen the intervals, but to calibrate them to current mix of production objects and monitor the reliability of this calibration.

•••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore