How to avoid breaking conversion learning and evaluation due to delayed labels?
In CVR, a user may convert minutes, days, or weeks after clicking, so labeling "as is" often turns future positive conversions into false negatives. This is critical for advertising, recommendations, marketplace funnels, and A/B tests, where the model is trained on incomplete logs.
Problem: label may not yet be available.
For a click at time
click_time = t, we want to estimate:P(conversion | click, x)But at the time the dataset is collected (T), we only know one of two things:
☞ Conversion has already occurred before T.
☞ Conversion is not yet visible.
The second case does not mean
converted = 0. It's a censored observation: the user was not observed for long enough.A typical mistake:
converted = 1, if conversion_time - click_time <= 7d
converted = 0, else
Fresh clicks have artificially low CVRs, the model learns from false negatives, offline metrics depend on the "maturity" of the data, and production calibration drifts: the model predicts the full-window CVR, while monitoring sees the partial-window CVR.
Baseline: Train only on mature data.
If the target horizon is a 7-day conversion, and data is available up to
2025-01-31, then for training, use clicks no later than 2025-01-24.➜ Pros:
☞ Honest labels
☞ Simple validation
☞ Easy to debug the pipeline and identify data leakage
➜ Cons:
☞ Loss of fresh data
☞ Poorer adaptation to seasonality and traffic changes
☞ With a long conversion lag, the training data becomes significantly outdated.
Practical advice: Explicitly store the
event_time, label_observed_until, horizon, and label_age fields in the feature store or training dataset. Without them, it's impossible to reproduce the labeling and understand why the CVR changed after retraining.A more robust approach: Model the delay.
The problem can be broken down into the probability of conversion and the distribution of the delay:
P(y = 1, delay <= H | x)For example:
☞ CVR model estimates the probability of conversion itself.
☞ delay model estimates
P(delay <= age | y=1, x)This approach is closer to survival analysis: there's an event, the time until the event, and censored observations. This approach is particularly useful if the delay depends on the product category, channel, geography, price, device, or retargeting strategy.
An alternative is a discrete hazard function:
P(conversion at day k | no conversion before day k, x)In this case, a click observed only 2 days ago is still useful for training the first two steps, rather than being discarded entirely. The trade-off is that the model and inference become more complex, but there's less data loss and a more accurate handling of the long tail of conversions.
Evaluation: The test set should also be mature.
If the
horizon = 7d, the holdout set should only contain objects for which at least 7 days have passed since the click. Otherwise, you're measuring the immaturity of the labels, not the quality of the model.A good approach:
train: clicks [D0, D1]
validation: clicks [D2, D3]
label cutoff: >= D3 + horizon
In the production environment, also consider metrics related to delay buckets:
* 0-1 hour
* 1-24 hours
* 1-3 days
* 3-7 days
* 7 days+
This helps identify where the system is failing: whether it's in fast conversions, the long tail, data freshness, attribution, or due to censored labels. This is especially important for A/B tests: an early readout can overestimate the effect of a model that performs well with fast conversions but underperforms over the full time window.
Conclusion:
Delayed feedback in CVR (Conversion Rate) is not just a matter of labeling; it's an engineering limitation of the ML system. Without mature labels, a clear label cutoff, and proper validation, the model optimizes for logging artifacts instead of actual conversions.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore