Netflix Priority-Based Load Shedding

In the previous blog post we already discussed cascading failures and way to address them. One of the approaches was load shedding and graceful degradation. Today we'll explore how these techniques are used in practice to improve user experience at Netflix.

In November 2020, Netflix introduced the concept of prioritized load shedding at the API gateway level:
✏️ Classify incoming traffic:
- NON_CRITICAL: This traffic does not affect playback or user experience (e.g., logs and background requests)
- DEGRADED_EXPERIENCE: This traffic affects user experience, but not the ability to play videos (e.g., stop and pause markers, language selection in the player, viewing history)
- CRITICAL: This traffic affects the ability to play.
✏️ Categorize the requests into priority buckets on API gateway level (Zuul)
✏️ Operate as usual under normal conditions
✏️ Drop lower priority requests if system is overload, higher priority requests still get served
✏️ Drop traffic progressively, starting with the lowest priority
✏️ Send a signal to clients to indicate how many retries they can perform and what kind of time window they can perform them in. Requests with higher priority will retry more aggressively than lower ones, also increasing streaming availability.

That approach helps to shed enough requests to stabilize services without members noticing the degradation, improving overall user experience.

In June 2024, Netflix published enhancement of their previous prioritized load shedding approach:
✏️ Add requests prioritization logic at the service layer additionally to the logic on API Gateway
✏️ Classify incoming traffic on service layer to the following buckets:
- CRITICAL: Affect core functionality. These will never be shed until full service failure
- DEGRADED Affect user experience. These will be progressively shed as the load increases
- BEST_EFFORT: Do not affect the user. These will be responded to in a best effort fashion and may be shed progressively
- BULK: Background work, can be shed
✏️ Categorize the requests based on the upstream client’s priority or other request attributes
✏️ Operate as usual under normal conditions
✏️ Drop lower priority requests if system is overload, higher priority requests still get served
✏️ Implement additional logic to support correct autoscaling triggers. Example: shed the requests only after hitting the target CPU utilization if an autoscaler is based on CPU metric

According to the article, priority-based load shedding helps to keep high availability of critical user features during multiple infrastructure outages: there were throttling more than 50% of all requests but the availability of user-initiated requests continued to be > 99.4%.

#systemdesign #reliability #usecase