How to avoid recalculating the same things in real-time?
In production recommendation systems and NLP services, generating embeddings is often a bottleneck. When a single user request requires inferring embeddings for dozens of candidates, each request requires a vector representation of each object. Latency increases, and specialists often overlook a simple optimization trick: caching within a batch, rather than globally.
➡️ Why batch-level caching?
Requests to the inference service arrive in batches. If you calculate the embeddings for each candidate from scratch, the number of calls to the model increases linearly with the number of requests and objects in each batch. However, many objects are repeated between requests: top-recommended products, popular texts, frequent entities. Batch-level caching solves this by deduplicating IDs within a single batch, minimizing redundant calculations.
➡️ How it works?
You collect all object IDs from all requests in the batch, deduplicate them, calculate embeddings only for the unique IDs, and then distribute the results through a mapping. Here's an example in Python:
def process_batch(batch_requests):
all_unique_ids = {}
for req in batch_requests:
for item_id in req['item_ids']:
all_unique_ids[item_id] = True
unique_ids_list = list(all_unique_ids.keys())
embeddings_map = compute_embeddings(unique_ids_list)
results = []
for req in batch_requests:
req_embeddings = [embeddings_map[i] for i in req['item_ids']]
results.append(compute_scores(req['user_vec'], req_embeddings))
return results
➡️ Production example:
Suppose you have a service for scoring ads in real-time: 100 requests in a batch, each with 50 items, but only 200 unique items. Without caching, there are 5000 calls to the embedding model; with batch-level caching, there are 200. That's a 25-fold reduction. For a latency-critical pipeline, this is the difference between an SLA violation and stable operation.
Practical advice and trade-offs
Add a second-level LRU cache for hot items with a TTL of 1 minute. Batch-level caching is the first filter, eliminating duplicates within the batch, while a global cache catches reuse between batches. But don't forget: synchronization is required within the pipeline, you need to collect all IDs before calculations. This can become a bottleneck for batch sizes greater than 1000 or with asynchronous processing. A common mistake is to confuse batch-level caching with a global TTL cache and miss duplicates within a single batch.
Conclusion: Batch-level caching is the simplest way to reduce duplicate load on embedding generation in real-time, reducing latency by orders of magnitude without significant overhead in terms of memory or infrastructure.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore