What Are LLM Evals? Types, LLM-as-a-Judge, and How to Build One (2026)

LLM evals are repeatable tests for language model systems: you run a model, a prompt, or an agent on a fixed set of inputs, score every output against criteria written down in advance, and compare the scores across versions. An eval turns "the new prompt feels better" into "the new prompt passes 87 of 100 cases instead of 74". Without one, every prompt edit and model upgrade is a guess that gets tested on your users.

The word covers two different things, and mixing them up causes most of the confusion. Benchmarks are public evals used to compare models: MMLU for knowledge, SWE-bench for fixing real GitHub issues, and research-task suites like the ones labs use to track recursive self-improvement. Application evals are the ones you build for your own product, from your own inputs, with your own definition of a good answer. Benchmarks help you pick a shortlist of models. Your own eval decides.

The parts of an eval

Every eval, from a spreadsheet to a CI job, has the same four parts:

1. A dataset. The inputs, ideally real ones from production or from the people who will use the system, plus any reference answers you have. Include the boring common cases and the failures you already know about.
2. The system under test. A specific prompt, model version, temperature, and, for agents, a specific agent harness. Pin all of it, or you will not know what changed.
3. A scoring method. Code checks, reference comparisons, an LLM judge, human review, or a mix (below).
4. A comparison. Scores only mean something next to other scores: the old prompt against the new one, model A against model B, this week against last week.

Ways to score an output

Pick the cheapest method that can actually tell good from bad for each criterion. Most useful evals combine several.

• Code-based checks. Is the output valid JSON that matches the schema? Does it contain the required field, stay under the length limit, avoid the banned phrase, compile, pass the unit tests? Deterministic, free to run, and impossible to argue with. Use them for everything they can cover.
• Reference comparison. Compare the output with a known good answer: exact match for classification and extraction, numeric tolerance for calculations, overlap or embedding similarity for text. Works when there is one right answer; misleading when there are many good ones.
• LLM-as-a-judge. A model grades the output against a rubric, or compares two outputs and picks the better one. Scales to qualities code cannot check, such as whether an answer is faithful to the source, specific, or follows the instruction. Needs calibration (below).
• Human review. Slow and expensive, and still the ground truth for anything subjective. Its best use is not scoring everything but labelling a sample that you use to check the automated scores.

Evals also run in two places. Offline evals run before a change ships, on a fixed dataset. Online evals score a sample of live traffic after it ships, which is how you notice drift when the inputs change under you.

LLM-as-a-judge, and how it goes wrong

Using a model to grade a model became the default for open-ended outputs after the 2023 MT-Bench and Chatbot Arena study (Zheng and colleagues) found that a strong judge agreed with human preferences more than 80% of the time, about the same rate at which human raters agreed with each other. The same paper documented the biases that still matter:

• Position bias. In pairwise comparisons, judges tend to favour whichever answer is shown first (or, for some models, second). Fix: run each pair in both orders, or shuffle and label outputs neutrally.
• Verbosity bias. Longer answers score higher even when they are not better. Fix: make length a stated criterion, and include short correct answers among your calibration examples.
• Self-preference. A judge tends to rate outputs from its own model family higher. Fix: use a judge from a different family than the systems being compared, or two judges from different families and look at where they disagree.

Two more problems are less discussed. Judges are weak at noticing what is missing: they grade the text in front of them and rarely ask what a complete answer would have included. And a vague rubric produces vague scores. "Rate the quality from 1 to 10" invites a 7 for everything; "Does the answer state the date range it used? yes or no" does not. Our coverage of Claude Code rewriting the Claude app noted the same gap from the other side: aggregate LLM-judge scores can look healthy while cross-file consistency failures slip through.

The working rules: narrow, mostly binary criteria; evidence before verdict (ask the judge to quote the part of the output that justifies each score); blind and shuffled outputs; a judge from another model family; and a check against 30 to 50 human labels before you trust the judge on its own.

A worked example: 256 outputs, one blind judge

Our sister site ran a study that shows what a small, honest eval looks like. The prompt structure study on promptbuilder.cc took 16 real tasks, wrote each one as four prompt structures (a bare ask, role plus context, a structured four-part prompt, and the structured prompt plus an example), and ran all of them through four models: 256 outputs in total. For each task and model, the four outputs were shuffled and labelled A to D so the judge never knew which structure produced which, then scored on four criteria (task fidelity, specificity, format, accuracy) from 0 to 5 each, in 64 judge calls.

The bare ask averaged 9.56 out of 20; the structured four-part prompt averaged 18.97. That gap is large enough to survive the method's weaknesses, and the study states them: one judge model, from the same family as one of the four models tested, which is exactly the self-preference risk described above. The more interesting result is per model. Adding an example helped two models and lowered Claude Sonnet 5's average by 1.94 points, the kind of finding you only get from an eval and never from trying a few prompts by hand.

How to build your first eval in an afternoon

1. Collect 30 to 100 real inputs. Pull them from logs, support tickets, or the requests your team actually makes. Add every failure you remember.
2. Write the criteria before you look at outputs. Three to five, each checkable. For an agent, include criteria about the process (did it report the error it hit?) as well as the result.
3. Automate what code can check. Schema, length, required fields, tests.
4. Add a judge for the rest, with the prompt below, and compare it to your own labels on a sample before trusting it.
5. Run it on every change. New prompt, new model version, new tool. Keep the scores in a file next to the prompt so the history is readable.
6. Refresh a slice every quarter. A fixed eval ages: models eventually ace it without getting better at your job.

For agents, final-answer scoring is not enough. A run can end with a confident "done" while a tool call failed halfway through, which is the subject of our silent agent failures guide; score the trace as well. Benchmarks have the same blind spot at a larger scale: open-source prompt injection detectors did well on synthetic tests and poorly against realistic agent attacks, and local models that look fine on coding benchmarks fall apart on multi-step agentic tasks. For teams testing agents without touching real systems, a world model such as Qwen-AgentWorld can stand in for the environment during eval runs.

An LLM judge prompt you can copy

You are grading one output from an AI system. You do not know which system or prompt produced it.
<task>[the original instruction or user request]</task>
<reference>[optional: a known good answer, or "none"]</reference>
<output>[the output to grade]</output>
Grade each criterion separately. For each one, first quote the part of the output that is relevant (or write "nothing relevant"), then give a verdict.
1. [criterion, e.g. "Answers the question that was asked, not a nearby one"]: pass or fail
2. [criterion, e.g. "Every number is traceable to the input or the reference"]: pass or fail
3. [criterion, e.g. "Follows the required format exactly"]: pass or fail
4. Missing: list anything a complete answer would include that this output omits.
Do not reward length. Do not guess about facts you cannot check from the input.
Return JSON: {"criteria": [{"id": 1, "evidence": "...", "verdict": "pass"}], "missing": ["..."]}

Binary verdicts with quoted evidence are easier to check against human labels than a 1 to 10 score, and the "missing" line pushes the judge toward the failure it is worst at noticing. Save the judge prompt with the eval, version it like code, and rerun the human comparison whenever you change it.