LLM Evaluation & Benchmark

When people say “LLM evaluation,” they often collapse five problems into one. That leads to weak evaluation setups. In practice, you are answering:

A lot of teams build an evaluation stack that only answers the first question. They score a final response, see a number go up, and assume the system improved. In practice, the model could have learned how to look better according to the metric. That is why evaluations are treated as a measurement system with failure modes.

LLM Evaluation Framework

1. Task Definition: “Answer user questions well” is not a task. “Answer user questions using only retrieved documentation and cite evidence” is a task. “Act as an agent that selects tools, makes API calls, and completes a workflow” is a task.
2. Define Success: In plain English write whether you are rewarding correctness, usefulness, speed, clarity, factuality, safety, low cost, or some mix. Without a clear definition of success, the evaluation will drift toward whatever is easiest to measure.
3. Choose the evaluation method: If the task has a narrow answer space, reference based metrics may help. If it is open ended, rubric based judging and human evaluation matter more. If it involves tools, you need trajectory level evaluation. If it requires factual grounding, you need claim verification.
4. Dataset design: A good eval set should include normal cases, hard cases, adversarial cases, ambiguous cases, and realistic production cases. Otherwise, your system will overfit to the happy path.
5. Reliability: You need to know whether the evaluation is stable across repeated runs, prompt paraphrases, changed answer order, and small formatting differences. If the score changes a lot under minor changes, you cannot trust it.
6. Failure analysis: Aggregate scores do not tell you what broke. You need to inspect clusters of failure: hallucination, missing steps, bad tool routing, policy over refusal, wrong citation, brittle formatting, shallow reasoning, and so on.

Kinds of LLM Evaluation

• Reference based evaluation: Compares model output to one or more reference answers. This includes BLEU, ROUGE, and METEOR.
• Human evaluation: Uses expert or non expert raters to score outputs according to a rubric. It is more expensive, but often closer to the real objective.
• LLM as a judge: Uses a model to apply a rubric, compare outputs, or classify failure modes. It is scalable and flexible, but carries bias and calibration problems.
• Task based evaluation: Measures whether the model actually completes the task. For code, this could mean test pass rate. For retrieval, it could mean answer correctness grounded in source documents. For agents, it could mean successful task completion under realistic tool constraints.
• Benchmark evaluation: Uses public datasets like MMLU, AIME, PIQA, SWE-bench, HarmBench, and TAU bench.
• Reliability evaluation: Asks whether the model or the evaluation method is stable. E.g. consistency across runs, judge agreement, and sensitivity to perturbations.

Reference based evaluation

Reference based evaluation compares the generated answer to a reference answer. They are cheap, fast, and easy to automate. The problem is that they measure similarity, not quality. That distinction matters. If there are many valid ways to answer a question, then overlap with a reference becomes a weak proxy. The model may produce a better answer than the reference and still score lower. Or it may memorize the reference style and score high while being less useful. This is why reference metrics work best when outputs are constrained and least when outputs are open ended. Common reference based evaluation metrics include:

Modern LLM tasks often involve multiple valid answers, hidden reasoning, real world knowledge, citations, tools, and safety constraints. Reference based metrics cannot see most of that. They do not know whether the answer followed the instruction exactly. They do not know whether a cited fact was hallucinated. They do not know whether a tool was selected correctly. They do not know whether the reasoning path was valid. They do not know whether the model stayed within policy. This is why older NLP metrics are still useful as narrow instruments rather than full evaluations. They can tell you about one slice of behavior, not the whole system.

Human Evaluation

Human evaluation remains one of the most important evaluation methods because humans can assess nuanced qualities that automatic metrics miss. They can notice whether an answer feels useful, whether the reasoning makes sense, whether the tone is appropriate, whether the response resolves the user’s actual need, and whether the factual claims feel trustworthy. But human evaluation is not automatically high quality. Humans disagree. Raters bring different standards, attention levels, domain knowledge, and biases. If the rubric is vague, the labels will be noisy. A good human evaluation setup uses a clear rubric, rater training, gold examples, adjudication for disagreements, and spot checks for quality. You also want enough redundancy to estimate inter-rater agreement. If three humans do not agree on what “good” means, then your evaluation criteria are underspecified. Human evaluation is also expensive and slow, so most teams only use human evaluation for calibration, benchmark creation, periodic auditing, and hard cases.

LLM as a Judge

LLM as a judge means using an LLM to score correctness, completeness, clarity, tone, instruction following, safety, and more. It works by comparing two candidate answers and choosing the better one. This makes it more flexible than BLEU or ROUGE, but creates new problems since the judge model brings its own biases, blind spots, and instability. So LLM as a judge is powerful, but it is not objective truth. It is a learned evaluator that must itself be evaluated.

For a detailed breakdown on when to use LLMs as judges, common biases, and structured outputs, please refer to our deep dive: LLM as a Judge Deep Dive.

Evaluation Dimensions

• Correctness: asks whether the answer is actually right. This is the core dimension for most tasks, but it can mean different things in different settings.
• Factuality: asks whether factual claims are true and grounded in available evidence. This is important for RAG, search, QA, and assistants that cite sources.
• Completeness: asks whether the answer covers the key parts of the task. A concise answer can still fail if it omits important elements.
• Instruction following: asks whether the model obeyed the requested format, scope, constraints, and tone.
• Relevance: asks whether the answer addresses the question instead of drifting.
• Clarity: asks whether the answer is understandable and well organized.
• Conciseness: asks whether the answer avoids unnecessary material.
• Reasoning quality: asks whether the logic is coherent, especially for multi step tasks. You must be careful here, since apparent reasoning can be nonsense.
• Safety: asks whether the model avoids harmful, disallowed, or risky outputs.
• Tool correctness: this includes choosing the correct tool, passing correct arguments, interpreting outputs correctly, and recovering from failures.
• Groundedness: matters when external evidence is present. A grounded answer ties claims to source material rather than inventing them.

How to Quantify Factuality

Factuality is one of the hardest evaluation dimensions because a fluent answer can feel true while being wrong. It also breaks simplistic judging. A judge may say something is factual because it sounds plausible. The better way is to decompose the problem into claims. First, extract atomic factual claims from the answer then evaluate each claim against evidence. Each claim can be marked as supported, unsupported, contradicted, unverifiable, or irrelevant. From there you can compute statistics such as the fraction of supported claims, contradiction rate, unsupported claim rate, and evidence coverage. This is much more informative than a single impressionistic “factuality score.”

When the task is grounded QA or RAG, you should require evidence based verification. The judge should only use the provided context, not its own background knowledge. Otherwise the judge itself may hallucinate validation. Citation level evaluation helps here. Ask whether each cited passage supports the associated claim. A response can cite a real source and still misrepresent it. For higher stakes tasks, combine LLM judgment with deterministic or external checks. For example, structured database lookups, exact answer matching on known entities, human expert review, or claim verification pipelines.

Benchmarks

• MMLU: (Massive Multitask Language Understanding) tests broad knowledge and academic style question answering across many subjects. It is useful for general reasoning and knowledge breadth, but not a measure of real world usefulness.
• AIME: (American Invitational Mathematics Examination) tests math reasoning. It is useful for hard symbolic reasoning, but is not everyday user tasks.
• PIQA: (Physical Interaction Question Answering) tests physical commonsense. It measures whether a model understands basic interaction with the physical world.
• SWE-bench: measures coding ability in realistic repositories by checking whether the model can produce patches that resolve actual issues. This is far more meaningful than code style scoring because it tests real task completion.
• HarmBench: evaluates safety behavior. It tests how the model responds in harmful or disallowed contexts and whether it resists unsafe instructions.
• TAU-bench: (Tool Agent User Interaction Benchmark) evaluates agents operating in environments with tools and user like interactions. This matters because tool use introduces many new failure modes that do not appear in QA benchmarks.

Reliability and Consistency Metrics

Most teams focus on “how high was the score?” and neglect “how stable was the score?” this is a mistake, because reliability is part of evaluation quality. A good evaluation remains consistent across runs, robust to prompt paraphrases and answer order, and stable under minor formatting noise unless formatting is relevant. It should also show reasonable agreement with human judgments on difficult or ambiguous cases. Useful reliability metrics include run-to-run variance, inter-judge agreement, agreement with human labels, order sensitivity, rubric adherence rate, and for structured outputs, schema compliance and missing field rates.

Analysis & Attribution

Failure Taxonomy & Slice Analysis

A judge model's true value emerges when you evaluate outputs across thousands of generations and cut the performance into slices. Did the model fail on Python specifically? Did it hallucinate API responses? Did it fail only on high-context prompts exceeding 16k tokens? This Slice Analysis is critical because it prevents regressions from hiding behind an average score increase.

Root Cause Attribution

Once a taxonomy of failures is identified via the judge model, you must perform Root Cause Attribution. Does the model hallucinate imports because the initial Scenario Design didn't include enough complex module resolutions? Is the Reward Model incorrectly scoring verbose but broken code? By attributing failures precisely back to the dataset or rubric, you close the iteration loop, dictating exactly what synthetic data must be generated in the next training cycle.

Agentic Evaluation

Agentic evaluation means evaluating how an agent makes decisions across time. It is selecting routes, calling tools, handling tool outputs, maintaining state, recovering from failures, and producing a final result. That means final answer scoring alone is not enough. An agent can arrive at a correct answer for bad reasons, or fail because of a small tool routing issue even if its general reasoning is sound. So agentic evaluation needs to examine the trajectory. You should evaluate whether the agent understood the task, chose the right subgoal sequence, selected the correct tool, used valid arguments, interpreted the output correctly, recovered from errors, and terminated appropriately. Each of these can fail independently. Agent systems also need environment aware evaluation. The same agent can look strong in a sandbox and fail in production because tools are slower, APIs return partial data, or the naming conventions are confusing.

How to Do Agentic Evaluation

The best starting point is task decomposition. Break a realistic user workflow into stages. For each stage, specify what success and failure look like. Then log the entire trace. This includes tool selection, tool arguments, tool outputs, retries, error handling, internal planning if available, and the final answer. If you only store the final answer, you lose most of the information needed to debug agent failure. Then define metrics at multiple levels. Final task success is one metric, but not the only one. You also want step success rate, tool selection accuracy, argument validity, tool output interpretation accuracy, recovery success rate, and unnecessary call rate. Then run the agent on realistic scenarios that includes ambiguous instructions, missing tools, flaky tools, confusing APIs, contradictory signals, and timeouts. That is where real failure modes appear. Then inspect failure clusters. If a tool name is ambiguous, many runs will fail the same way. If the API naming is inconsistent, the agent will repeatedly call the wrong endpoint.

Common Agent Failure Modes

• Route Error: A route error happens when the agent chooses the wrong path or subworkflow. For example, searching instead of retrieving from memory, summarizing instead of verifying, or planning when direct execution would suffice. Route errors often come from ambiguous instructions, weak tool descriptions, or poor prompt scaffolding. The fix is clearer routing criteria, stronger tool documentation, better examples, and route specific evaluation.
• Using a Tool That Does Not Exist: Agents sometimes hallucinate tools or function names. This happens when the tool space is large, naming is inconsistent, or the model relies too much on prior patterns rather than the actual available tool list. The fix is to provide the available tools, keep names simple, enforce tool validation, and return errors when a nonexistent tool is attempted. Training and evaluation should include cases where the correct behavior is to not call a tool.
• Instruction Unclear: If the user’s request is ambiguous, the agent may choose a poor strategy. This is not always a model failure. Sometimes the system failed to define when to ask clarifying questions, when to make reasonable assumptions, and when to stop. The fix is better instruction policy and evaluation criteria. Test cases should cover ambiguity. The agent should be judged on whether it handled ambiguity appropriately, not just whether it guessed right.
• API Naming Is Not Logical: Many agent failures are really interface failures. If one API is called lookup_user, another is called fetch_profile, and a third is called get_person, the agent has to infer semantics from weak clues. Humans struggle with bad API design too. Models struggle even more. The fix is better tool ergonomics. Use semantically clear names, concise descriptions, typed schemas, example calls, and consistent argument conventions.
• Tool Call Error: No Response: A tool call that returns no response is a crucial edge case. A weak agent may treat silence as success. Another may loop forever. Another may crash. A stronger agent detects the failure, distinguishes between timeout and empty valid response, retries when safe, and escalates or changes strategy when needed. Evaluation here should not just mark final success or failure. It should ask: did the agent detect the no response condition, did it retry appropriately, did it avoid infinite loops, and did it communicate uncertainty properly? The fix is both behavioral and infrastructural. Tools should return error states where possible. Agents should have timeout awareness, retry policies, and fallback logic. Evaluation datasets should include no response cases.

How to Fix Agent Failures