The model didn't become smarter.
It just happened to see the correct answers in advance.
➡️ Let's break it down in 4 minutes, you'll understand where data leaks hide.
1. Data Leakage
Data leakage occurs when information that won't be available at the time of actual prediction is used during the model training process.
Because of this, metrics on the validation stage can look much better than the actual quality of the model on new, previously unseen data.
2. Model Evaluation
The test set isn't just "additional data".
It's a simulation of the future.
Only train the model on the information that would have been available to you at the time of prediction.
Evaluate it on examples that the model couldn't have influenced during training.
3. Direct Leakage
This is the most obvious type of leakage.
Examples:
- a field with information from the future;
- an ID that encodes the target variable;
- a variable that appears only after an event has occurred;
- duplicate records in both the training and test sets.
If a feature doesn't exist at the time of inference (prediction), then it's likely a source of data leakage.
4. Indirect Leakage
This is the type of leakage that most often traps teams.
You perform normalization, imputation, feature selection, outlier removal, or dimensionality reduction before splitting the data into a training and test set.
The model didn't directly see the data from the test set.
But your preprocessing pipeline already saw it.
5. Train/Test Split:
Wrong:
fit the scaler on all data → split the data → evaluate
Right:
split the data → fit the scaler only on the training set → apply it to both the training and test sets
The same idea applies to imputers, encoders, feature selection, PCA, and any preprocessing step that is trained on the data.
6. Cross-Validation:
Each fold is a mini-experiment with a training and test set.
Therefore, preprocessing should be performed within each fold.
If you prepared the entire dataset once and then ran cross-validation, each fold would already have had access to its held-out data.
7. Pipelines:
A pipeline isn't just a way to make the code cleaner.
It's also a defense against data leakage.
Combine preprocessing, feature selection, and the model into a single pipeline, and then pass this pipeline to cross-validation or hyperparameter search (grid search).
8. AI Engineering Version:
Data leaks also occur in RAG systems and when evaluating LLMs.
Leakage occurs when you tune chunks, prompts, re-rankers, thresholds, or examples on the same evaluation dataset that you later present as "held-out".
As a result, your benchmark turns into training data.
9. Leakage Checklist:
Before trusting the obtained metric, ask yourself:
Could this feature exist at the time of prediction?
Was any transformation (transform) step trained (fit) on the test data?
Did cross-validation include the entire pipeline?
Were we tuning parameters on the final evaluation dataset?
If the answer is "yes", then the metric likely doesn't reflect the actual quality of the model.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore