Temporal leakage in the feature store: How point-in-time joins, backfills, and feature causality checks save a model from beautiful offline metrics and failure in production?

Temporal leakage in the feature store is one of the most expensive ways to get great offline metrics and a useless model in production. The problem isn't that the feature is bad, but that on train it knows more than the model would have known at the moment of decision-making.

We predict churn on date t, but in the features we use transactions_last_30d, calculated after a backfill from a table where transactions arrived with a delay or were recalculated with future fixes.

Offline is all beautiful. Online - a slump.

1️⃣ Point-in-time join - basic protection

For each training row, there is prediction_time. The features should be in the state they were in at that moment.

It's important to distinguish:

- event_time - when the event actually happened;
- ingestion_time / created_at - when it entered the system;
- available_at - when the feature became available to the model;
- prediction_time - the moment of prediction.

The correct join should take into account not only event_time <= prediction_time, but also available_at <= prediction_time:

WITH ranked_features AS (
    SELECT
        l.entity_id,
        l.prediction_time,
        f.feature_value,
        ROW_NUMBER() OVER (
            PARTITION BY l.entity_id, l.prediction_time
            ORDER BY f.event_time DESC
        ) AS rn
    FROM labels l
    JOIN features f
      ON f.entity_id = l.entity_id
     AND f.event_time <= l.prediction_time
     AND f.available_at <= l.prediction_time
)
SELECT *
FROM ranked_features
WHERE rn = 1;

If there is no available_at, you often can't prove that there is no leakage.

2️⃣ Backfills - a hidden source of leakage

Backfills are dangerous because they create the illusion of historical completeness.

For example, today you recalculated a feature for the past year:

- corrected old events;
- added data from a new source;
- changed the business logic;
- caught up with late-arriving events;
- used a reference that wasn't available at the time.

As a result, train gets a history that didn't actually exist at the moment of prediction.

A correct backfill should answer the question:

What feature would the model have seen then if the pipeline had worked with the same delays, sources, and availability rules?

If the answer is unknown, it's not historical truth, but reconstructed truth. For model training, these are different things.

3️⃣ Checking the causality of features

Before training, every feature should be run through a causality review.

➡️ Minimum checklist:

1. Is the feature available before prediction_time?
It's not that the event happened, but that the value of the feature was available.

2. Is there a label proxy in the feature?
For example, days_since_last_payment_failed for a default task might be almost a direct consequence of a future target.

3. Is the aggregation window strictly in the past?
last_7d should mean [t-7d, t), not a calendar week that includes the future relative to t.

4. Are there future-aware reference tables?
Segments, statuses, limits, antifraud flags, and CRM attributes are often backfilled.

5. Is the source latency taken into account?
If the data arrives in 6 hours, you can't use an event at 09:55 for a prediction at 10:00.

In production ML, a feature is considered valid not when it's historically correct, but when it's demonstrably available to the model at the moment of decision-making.

••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore