In embedding-based systems, the encoder is part of the data contract. It cannot be updated like a regular ML model: the ANN index already contains vectors from the old space, and a common mistake is to assume compatibility due to the same dimension and metric.
Why compatibility breaks?
Even if the dimension is the same and the cosine is the same, and the offline benchmark is better, the new encoder does not have to be compatible with the old index.
After the update, the following change:
- geometry of the space;
- distribution of norms;
- local neighborhoods;
- ranking of nearest neighbors;
- calibration of scores;
- behavior of the ANN structure: HNSW/IVF/PQ were built for the old distribution.
The main anti-pattern: writing new documents with the new encoder into the old index with old embeddings.
Such an index becomes mixed: some vectors live in one space, others in another. The ANN works formally, but the nearest neighbors no longer have correct semantics.
Versioning the embedding space as a production contract
You need to version not just the
model_name, but the full contract:embedding_version = encoder + tokenizer + pooling + normalization + dim + metric
If any of these has changed, it's a new version of the space.
Practical advice: keep the embedding_version next to the document, query, index, and retrieval logs. Otherwise, if recall or CTR degrades, you won't understand which encoder was actually involved in the delivery.
Raising a new index and enabling dual-write
The old path:
docs_v1 -> embeddings_v1 -> ann_index_v1
The new path:
docs_v2 -> embeddings_v2 -> ann_index_v2
Even if the documents are the same, the embeddings must be recalculated with the new encoder. For ANN, this is a new corpus.
Importantly: the index parameters should also be tuned. For example, for HNSW, the old M, efConstruction, efSearch may not be optimal for the new distribution.
During the migration, write new and updated documents to both versions:
on_document_upsert(doc):
emb_v1 = encoder_v1(doc)
emb_v2 = encoder_v2(doc)
index_v1.upsert(doc.id, emb_v1)
index_v2.upsert(doc.id, emb_v2)
This is more expensive in terms of compute and ingestion latency, but the old retrieval continues to work and the new index catches up with the current state. If v1 is soon shut down, dual-write can be kept only until the cutover plus a short rollback window.
Backfill, shadow-read, and readiness criteria
For v2, we need to recalculate the embeddings of the entire corpus and upload them to the new index. Here, it's not about notebook metrics, but about engineering reliability:
- idempotency of tasks;
- control of lag;
- deduplication of upserts;
- checkpoints;
- separate limits on encoder and ANN ingestion;
- document count comparison between indexes;
- percentage of documents without v2 embeddings.
The migration is not ready until the new index covers the production corpus with an acceptable lag.
Before switching, enable shadow-read:
query -> encoder_v1 -> index_v1 -> results_v1
-> encoder_v2 -> index_v2 -> results_v2
Show only v1 to the user, but compare:
- recall@k on labeled data;
- overlap@k between v1 and v2;
- NDCG/MRR if there are clicks or raters;
- p95/p99 latency;
- tail failures;
- score distribution;
- downstream metrics in ranking, recommendations, or RAG.
Warning: high overlap@k does not guarantee product improvement. The new retrieval may change diversity, freshness, coverage, and load on the next ranker. It's better to do the cutover via a feature flag, with monitoring of quality, latency, error rate, and a quick rollback to
ann_index_v1.Conclusion: Updating the encoder is a migration of the embedding contract and ANN infrastructure, not a simple model replacement in the inference path.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore