Uber’s Gen AI On-Call Copilot

GenAI continues its march in routine automation. This time Uber shared their experience with Genie - on-call support automation for internal teams.

The issue is very common for large companies with many teams: there is some channels (for Uber, it's slack with ~45 000 questions per month) where teams can put questions and request help with the service or technology. Of course, there are a lot of docs and relevant articles, but they are fragmented and spread across internal resources. It's really hard for users to find answers on their own. As a result, the number of repetitive questions grows, load and demand on support engineers increase.

Key elements of implemented solution:
✏️ RAG (Retrieval-Augmented Generation) Approach to work with LLM
✏️ Data Pipeline: Information from wikis, internal Stack Overflow, and engineering docs is scraped daily, transformed into vectors, and stored in an in-house vector database with the source links. Data pipeline is implemented on Apache Spark.
✏️ Knowledge Service: When a user posts a question in Slack, Genie’s backend converts it into a vector and fetches the most relevant chunks from the vector database.
✏️ User Feedback: Users can rank answers as Resolved, Helpful, Not Helpful, or Relevant, these ratings are used to analyze answer quality.
✏️ Source Quality Improvements: There is a separate evaluation process to improve source data quality. The LLM performs docs analysis and returns an evaluation score, explanations of the score and actionable suggestions to improve. All these information is collected to an evaluation report for further analysis and fixes.

Since Genie’s launch in September 2023, Uber reports it has answered 70,000 questions with 48.9% helpfulness rate, saving 13 000 engineering hours😲. It's impressive! I definitely want to have something similar at my work. Just a small hurdle left—get the budget and resources for implementation. No big deal, right? 😉

#engineering #usecase #ai