Agent Trajectory Evaluation

Two agents can both deliver the right answer and still be very different products. One took three steps; the other took twenty-six, called a paid API eleven times, and deleted a file on the way. Grading the final answer cannot tell them apart. Agent trajectory evaluation grades the path instead of the destination.

What is being scored
The trajectory is the full ordered sequence of reasoning steps, tool calls, arguments and observations a run produced. Evaluating it means asking questions about the sequence itself:

Was the right tool selected at each decision point, with sensible arguments?
How many steps were needed, and was any work repeated or looped?
Did a failed call get handled, or retried blindly until the budget ran out?
Was any irreversible action taken that should have paused for approval?

Scores come from comparison against a gold reference path where one exists, from programmatic checks over the step sequence, or from LLM-as-judge rubrics applied to the recorded trace.

Why outcome metrics are not enough
Task success rate is one bit per run. That makes it nearly useless for debugging and easy to flatter: an agent can stumble onto a correct answer by a wasteful or unsafe route, and it will not stay lucky in production. Trajectory evaluation is also what turns a benchmark into a diagnosis — on SWE-bench or GAIA the pass rate tells you where you stand, while the trajectories tell you which step to fix. It depends entirely on having the steps recorded, which is why tracing is a prerequisite rather than a companion.

Agent trajectory evaluation scores the sequence of steps an agent took, not only whether the final answer was right. It examines tool choices, redundant or missing actions, step count, and recovery from errors. Trajectory-level scoring catches agents that reach correct outcomes by luck or at absurd cost, and it localises failure to a specific decision — information a single pass/fail verdict on the output cannot provide.

Picture marking a maths exam. Two students both write 42 at the bottom of the page. One got there in four clean lines. The other scribbled over three pages, made two errors that cancelled out, and landed on it by luck. The same mark hides a real difference.

Agent trajectory evaluation is marking the working, not just the answer, for software doing a job on its own. It looks at every step taken: which tools it picked, how many attempts it needed, whether it went round in circles, whether it did anything it should have asked permission for first. A right answer reached by a bad route will not keep being right.