You have 80000 agent trajectories from production.
You need to select those that should be sent for review to improve the agent.
It's not possible to use LLMs to evaluate the trajectories.
➡️ How would you solve?
The simplest option is random sampling. We take 100 random trajectories and send them for manual review. But in production, agents usually handle typical requests well, so a significant portion of the annotation budget is wasted on noise.
Another approach is to filter long dialogs, assuming that 10+ user messages indicate a more complex scenario. But long dialogs are heavily skewed towards obvious failures. In the end, you find obvious failures and miss subtle problems in scenarios where the agent formally succeeded.
A recent work from DigitalOcean proposes a different approach: calculating light behavioral signals directly from the trajectory data using deterministic rules on top of dialogs and execution logs.
The signals are divided into three groups:
1. Interaction signals - from the user-agent dialogue.
If the user reformulates the same request or corrects the agent - this is a desynchronization. If the agent repeats without progress - it's stagnation. A user's confirmation that everything worked - satisfaction. All of this is determined through a normalized comparison of phrases and checking the similarity of neighboring replies.
2. Execution signals - from tool calls and runtime events.
A tool call that returns an empty result or does not advance the task is considered a failure. Repeated calls with identical or "drifting" inputs indicate a loop. These signals are easily extracted from structured execution logs.
3. Environment signals - frequency limits, context overflow, API errors.
Useful for diagnostics, but not suitable for learning, as they reflect system limitations rather than the agent's decisions.
Each trajectory is assigned a score based on the signals that triggered, after which the trajectories with the highest score are sent for review.
On the τ-bench, three approaches were compared on 100 trajectories:
- Random sampling - 54% informative
- Heuristic by length - 74%
- Signal approach - 82%
That is, approximately 4 out of 5 selected trajectories are really useful for improving the agent.
Even among dialogs where the agent correctly performed the task, the signal approach found useful patterns in 66.7% of cases compared to 41.3% for random sampling.
Hidden PROBLEMS: policy violations, inefficient use of tools, unnecessary steps. The task is formally completed, but there is potential for optimization.
Entire pipeline works without overhead of LLMs and can continuously operate in production, labeling each trajectory at input.
If you need a practical implementation, this approach is already integrated into Plano - an open-source proxy for AI that combines routing, orchestration, protective restrictions and observability.
GitHub, Article
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore