Pinterest Ad System Redesign

A recent article on the Pinterest Engineering blog, titled "Redesigning Pinterest’s Ad Serving Systems with Zero Downtime," discusses the complete overhaul of Pinterest's recommendation system.
The ad-serving system is crucial to Pinterest's business, serving as the primary revenue channel. The previous system, known as "Mohawk," was implemented in 2014. However, as the company grew, Mohawk accumulated technical debt and complexity, making incident troubleshooting and bug analysis increasingly challenging.

The following architectural issues were identified as motivations for the reimplementation:
📍High coupling between business and infrastructure logic
📍Lack of modularization and ownership: Code from the same packages and files was owned by different teams.
📍No guarantees of data integrity
📍Unsafe multi-threading: Lack of error handling and potential race conditions.

The new design principles include:
📍Extensibility: Enable the addition of new features and the deprecation of old ones.
📍Separation of Concerns: Organize logic into separate modules, each owned by the appropriate teams.
📍Safety: Ensure the safe use of concurrency and enforce data integrity rules.
📍Development Velocity: Provide well-supported development environments and facilitate easy testing and debugging.

The team spent 2 years implementing the new version of the recommendation system - AdMixer. Rewritten in Java, like other Pinterest services, this new system helps unify the technological stack. It has been in production for three quarters now without significant outages. Additionally, it was designed to run on AWS Graviton instances (ARM architecture) to reduce AWS usage costs (more about that in Multi-Arch Images post ).

What I can say is that a complete system rewrite is a very expensive endeavor. It would be interesting to know the costs of this two-year project, as not all companies can afford such an exercise. An eight-year lifespan is not particularly long for a software system, indicating that serious architectural mistakes and unresolved technical debt prevented the previous system from evolving effectively.

Let's be honest: engineers often prefer to rewrite a system, especially one created by others😀. It’s fun, cool, and easier, and it makes a nice addition to a resume. However, it is far more challenging to build a system with the right architectural principles that support evolutionary rather than revolutionary development (do not confuse it with over-engineering!).

In any case, the Pinterest engineering team did a great job: they implemented a new approach, successfully delivered it to production, accelerated business features delivery, and increased developers satisfaction. I hope the new service incorporates lessons learned and it will be extensible for future growth and business needs without requiring a full rewrite.

#architecture #usecase #refactoring