Agent Trajectory Evaluation
An evaluation discussion about an agent that mostly reaches the right answer and that nobody on the team trusts.
The questions in this round
Your agent reaches the right final answer on most runs, but you do not trust it. How do you evaluate it properly?
You grade the tool calls too. On one case the agent calls the refund tool, gets an error, retries four times, then answers correctly. Pass or fail?
Same case, run ten times: seven pass, three take a different path and fail. What do you report?