LLM-as-Judge (Agent)
Checking whether an agent did its job sounds simple until you try to write the check. There is no single correct string to compare against: the agent took a path, made choices, and produced work a person would call good or sloppy. LLM-as-judge hands that verdict to another language model, given a rubric and the material to score.
How the judging works
The judge model is prompted as an evaluator and receives:
- the task the agent was given, so it knows what success means;
- the output to score — an answer, a patch, a report — and often the full trajectory of steps and tool calls that produced it;
- a rubric: the criteria, the scale, and what each point on it means;
- an instruction to justify before scoring, which makes the verdict auditable and steadier.
Verdicts come back as pass or fail, a rating, or a preference between two candidate runs. Agent leaderboards and suites such as Tau-bench lean on judged scoring exactly where no assertion can be written.
Why it matters
Without a judge, evaluation shrinks to what a machine can check exactly — tests that pass, a number that matches — and agents get tuned for that narrow slice while open-ended work goes unmeasured. The cost is that the ruler has opinions: judges reward length and confident phrasing, favour text resembling their own model's style, and shift when the judge model is upgraded. Treat it as an instrument. Pin the judge model and version, calibrate against human labels on a sample, and report the agreement rate alongside the scores.
LLM-as-judge is the use of a language model, prompted with a rubric, to score an agent's output or trajectory where no exact answer exists to compare against. It turns subjective quality — was the plan sensible, is the report accurate — into a usable metric, making open-ended agent work measurable at scale. Its verdicts carry biases toward length and style, so the judge model and version are pinned and calibrated against human labels.
How do you mark homework when there is no answer key? That is the problem with judging software that writes reports, fixes code, or plans a trip. There is no single right answer to check it against.
One practical trick is to ask another AI to be the marker. It is given the original task, the work that was produced, and a short list of what counts as good — then it explains its reasoning and awards a score. That makes it possible to grade thousands of attempts cheaply, instead of asking people to read them all.
The catch is that the marker has its own habits. It tends to be impressed by long, confident-sounding answers, so its scores are regularly checked against what real people think.