How to Avoid Latency?
Interpretability in production is great, but it becomes problematic when you calculate SHAP values for a batch of thousands of objects and experience latency of tens of milliseconds. With 10,000+ requests per second (RPS) and a time budget of less than 10 milliseconds, batch SHAP or LIME simply won't work. The main mistake is trying to calculate full attributions for every request without considering the trade-offs between accuracy and speed.
➡️ Three Approaches to Online Attribution
The first approach is TreeSHAP. It's the most accurate, but has a complexity of O(T*D*L). For CatBoost with 500 trees and a depth of 8, this already results in 100-200 microseconds per object. You can cache path-dependent gradients, but it's still computationally expensive.
The second approach is Fast SHAP approximation using expected gradients or Gradient SHAP. It works in O(T*D) – an order of magnitude faster. You lose some accuracy, but for most production tasks, the difference is not significant.
The third approach is surrogate LIME. You build a linear model on the fly around the request, using a sample of 100-200 objects. The time complexity is O(k*T*D), and it can be parallelized across rows.
➡️ How to Control Latency?
The most reliable method is adaptive timeout:
class OnlineAttributor:
def __init__(self, model, latency_budget_ms=5):
self.model = model
self.budget = latency_budget_ms / 1000
async def get_attribution(self, features):
shap_values = await asyncio.to_thread(
self._tree_shap, features, timeout=self.budget
)
if shap_values is None:
shap_values = await asyncio.to_thread(
self._global_importance, features
)
return shap_values
It's also helpful to:
* Batching: Group requests into batches of 10-50 and calculate SHAP values vectorially. The latency per item decreases significantly.
* Precomputed SHAP for streaming requests: Perform offline attribution every 10 minutes and cache the results. If the features don't drift drastically, this is often sufficient.
➡️ Typical Mistakes & Trade-offs
Linear models like LIME don't work well with the non-linearities of boosting. For CatBoost or LightGBM, it's better to use the built-in TreeSHAP through predict with pred_contrib=True – it's cheaper and more accurate.
Another common mistake is not considering the distribution of latency. With high throughput, the average latency might be 1 millisecond, but the 95th percentile could be 50 milliseconds due to complex objects. You need to set a timeout for the 99th percentile and fall back to global importance if the timeout is exceeded.
In my experience, a combination of Fast TreeSHAP with pruning, adaptive timeout, and fallback to global importance provides a typical time of less than 2 milliseconds per object with 100 trees. Without this, interpretation in production becomes a bottleneck.
➡️ What is the Conclusion? For online attribution in gradient boosting with high-throughput inference, use Fast TreeSHAP with adaptive timeout and caching, rather than full SHAP for every request. This provides a balance between accuracy and latency.
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore