How to Find Harmful Training Examples Before Fine-Tuning: Influence Functions, TracIn, and Data Pruning in Production ML

In production ML, "bad" training examples can be costly: a cluster of mislabeled, outdated, or anomalous objects can consistently degrade fine-tuning on a fresh data set. A common mistake is to clean the dataset only based on heuristics and not checking which samples actually increase the loss on the production-like validation set.

1️⃣ Influence Functions

Idea: estimate how the loss on the validation set z_val would change if we slightly increase the weight of the training example z_train.

I(z_train, z_val) ≈ - ∇L_val^T H^-1 ∇L_train

where H is the Hessian with respect to the model parameters.

If the influence is large and positive, the training example is likely harming the validation loss and quality.

Pros:
- rigorous theoretical formulation;
- can associate specific training examples with specific model errors.

Cons:
- expensive H^-1;
- poorly scalable to large neural networks;
- sensitive to non-convexity, batchnorm/dropout, checkpoints, and Hessian approximation.

In production, we typically use approximations: LiSSA, conjugate gradients, low-rank approximation, or calculate the influence only for the last layer/head of the model.

2️⃣ TracIn

A more engineering-oriented approach: a training example is useful for the validation set if their gradients evolve similarly during training. It's harmful if they evolve in the opposite direction.

TracIn(z_train, z_val) = Σ_c η_c · ∇L_train(θ_c) · ∇L_val(θ_c)

where θ_c are checkpoints and η_c is the learning rate.

A strongly negative score means: the training example is pulling the model in the opposite direction of what's useful for validation.

A mini sketch for the last layer:

for ckpt in checkpoints:
    model.load_state_dict(load(ckpt))

    g_val = mean_grad(model.head, val_loader)

    for i, batch in enumerate(train_subset):
        g_train = grad(model.head, batch)
        scores[i] += lr[ckpt] * dot(g_train, g_val)

harmful = argsort(scores)[:K]

Practical advice: calculate the score not on the entire validation set, but on important production slices: new users, rare classes, problematic regions, fresh drift, segments with high business value or SLA.

3️⃣ Data pruning before fine-tuning

Workflow:

1. Freeze a production-like validation set without leakage.
2. Train a baseline / fine-tune and save several checkpoints.
3. Calculate the influence or TracIn for train→val.
4. Check the top harmful samples:
- label noise;
- outdated distribution;
- conflicting duplicates;
- corrupted inputs;
- incorrect task/schema version.
5. Remove, downweight, or relabel them.
6. Repeat fine-tuning and check not only the overall metric but also the regression by segment.

Production example: before retraining a recommendation model on fresh logs, you might find old interactions with a changed product taxonomy, conflicting labels after a schema migration, or bot traffic that degrades the ranking loss on a fresh holdout.

4️⃣ Caution

Don't blindly remove all "harmful" examples. Sometimes they degrade the current validation, but are needed for long-tail robustness, fairness, or resilience to rare scenarios.

It's safer to start with top-K, do a human-in-the-loop audit, compare remove / downweight / relabel options, and look at the trade-off between quality, latency of recalculation, cost of labeling, reproducibility, and monitoring reliability.

Conclusion: Influence Functions and TracIn are useful not as a magic data cleaning tool, but as an engineering approach to make fine-tuning less toxic to noise, outdated data, and conflicting labeling.

••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore