In RAG retrieval, things often break silently: same model, same embedding model, same prompt, normal latency, but the answers have gotten worse.
A typical mistake is to immediately tweak the prompt or blame the LLM, even though the problem lies deeper: the corpus has changed.
1️⃣ Monitor the corpus drift itself
We don't directly measure quality, but we look at how the space in which the retriever operates has changed:
- distribution of embedding chunks;
- average chunk length, overlap, number of chunks per document;
- proportion of new, deleted, and modified chunks;
- duplicates and near-duplicates;
- distribution of domains, document types, languages, dates;
- density of the embedding space: have many chunks "clumped" together.
If the corpus has noticeably shifted, old retrieval thresholds and expectations of top-k might become garbage. Especially if the confidence logic is tied to score or the gap between top-1 and top-2.
2️⃣ Anchor queries instead of labels
In production, there are almost never labels like "these chunks are relevant for this query". But we can take a stable set of production queries: for example, 500-5,000 frequent or business-critical queries.
This isn't annotation. We don't know the correct chunk. But we know that the retrieval behavior shouldn't change chaotically after each corpus update.
For each anchor query, save the baseline:
- top-k doc/chunk ids;
- retrieval scores;
- rank positions;
- gap between top-1 and top-2;
- diversity of top-k;
- source distribution.
After the corpus update, compare the new retrieval with the baseline.
Useful proxy metrics:
- Jaccard@k between the old and new top-k;
- p95_top1_score_drop;
- score_wasserstein between the baseline and current scores.
3️⃣ How to interpret the signals
- mean_jaccard@10 has dropped sharply: the retriever has started returning different context;
- the top-1 score systematically drops: the queries are matching the corpus less well;
- the score distribution has shifted significantly: old thresholds and confidence logic might have broken.
Practical advice: don't just look globally, but also by segments - sources, languages, document types, product domains. A global average easily hides degradation in a critical segment.
4️⃣ Retrieval confidence without ground truth
Even without annotations, you can look at the "confidence" of the retriever:
- high top-1 score;
- large gap between top-1 and top-2;
- consistency of dense retrieval and BM25;
- stability of top-k when query rewriting;
- low proportion of duplicates in top-k;
- coverage of needed sources.
If dense and lexical retrieval suddenly start diverging, don't just chalk it up to noise. Often, this means that the corpus or queries have changed in a way that one of the strategies no longer works as before.
Production minimum for RAG:
- store a snapshot of retrieval results for anchor queries;
- calculate overlap, score drift, and rank churn after each corpus update;
- monitor duplicates, new chunks, and source distributions separately;
- set alerts not on a single query, but on aggregates by segments.
Corpus drift is annoying because it doesn't look like a crash. The system responds, there are no errors, and the latency is normal. It's just that the context has become slightly less relevant. Then a little more. And the RAG quality slowly declines.
The conclusion is WITHOUT LABELS, you CAN'T honestly measure relevance, but you can monitor the stability of retrieval behavior, the retriever's confidence, and corpus changes to catch degradation before users do.
•••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore