AI deliveryHafsteinn Runarsson · AI Konsulent11 Aug 2026 · 7 min
How to evaluate LLM applications: a framework
To evaluate LLM applications well, test more than the model in isolation. A useful application combines the model with prompts, retrieval, tools, business rules, and an interface. The evaluation has to cover that whole system and the job it is supposed to do.
A strong evaluation answers a practical question: can this application help the intended user complete a defined task, within acceptable limits, often enough to justify putting it into production? The answer should come from repeatable test cases rather than a handful of impressive demos.
- Define the job before the score
Start with a narrow statement of what the application should do. Name the user, the task, the source material it may use, and the action it may take. "Answer questions about our policies using approved documents" is testable. "Be a helpful company assistant" is not.
Write down the boundaries as carefully as the happy path. What must the application refuse? When should it ask for missing information? Which actions need confirmation? What should happen when a source is unavailable or two sources disagree?
This scope becomes the evaluation contract. It prevents the team from changing the definition of success after seeing the output.
- Turn the job into observable outcomes
Choose outcomes that can be checked. For a support copilot, that might be selecting the right policy, giving the user a usable next step, and escalating when the request falls outside its authority. For a document workflow, it might be extracting required fields, preserving exact values, and routing exceptions to a person.
Separate outcomes by severity. A slightly awkward sentence is not equal to inventing a policy or taking an unauthorized action. A simple severity scale keeps the total score from hiding rare but serious failures.
Use three categories:
Pass: the user can complete the task without correction.
Needs review: the answer is usable, but a person must fix or verify part of it.
Fail: the answer is wrong, unsafe, unsupported, or blocks the task.
You can add more detail later. Begin with labels that reviewers can apply consistently.
- Build an evaluation set that resembles real work
An evaluation set is a collection of inputs, context, expected behavior, and scoring instructions. Build it from the work the application will actually encounter. Remove personal or confidential data before storing examples in a reusable test set.
Include ordinary requests, ambiguous requests, missing context, conflicting context, malformed inputs, and requests the application should refuse. If the application uses tools, include tool failures, empty results, stale records, and permission errors. If it retrieves documents, include cases where the correct answer is absent.
Do not fill the set with near-duplicates. Each case should test a distinct behavior or risk. Give every case a stable identifier so a failure can be tracked across versions.
For open-ended tasks, write a rubric instead of one ideal answer. The rubric should say what must be present, what is optional, and what causes an automatic failure. Reviewers can then judge different but equally valid answers without forcing the application to copy a reference response.
- Evaluate the application in layers
One score is rarely enough to diagnose a problem. Score the system in layers so the team can see where it failed.
Task completion asks whether the user reached the intended outcome.
Correctness checks whether names, values, calculations, classifications, and instructions match the supplied evidence or expected result.
Grounding checks whether factual statements are supported by the sources available to the application. A grounded answer should also say when the sources do not contain the answer.
Instruction following checks whether the output respects the requested format, language, tone, and workflow rules.
Robustness checks whether small changes in wording, order, or irrelevant context change the result in ways that matter.
Safety and control checks refusals, permission boundaries, confirmation steps, and handling of sensitive data.
Operational quality covers response time, cost per completed task, tool reliability, and the rate of requests that need human intervention. Set thresholds from the needs of the workflow rather than copying someone else's benchmark.
These layers should lead to action. A grounding failure points toward retrieval or citation changes. A task-completion failure with correct facts may point toward the workflow or interface. A slow but accurate result may need a different model, fewer tool calls, or a simpler path.
- Combine deterministic checks, review, and model-assisted scoring
Use deterministic checks wherever the requirement has a clear answer. Schema validation, required fields, exact values, allowed tools, citation presence, and permission rules can often be checked automatically. These checks are fast to rerun and easy to explain.
Use human review for judgment that depends on context, trade-offs, or domain expertise. Give reviewers the same rubric and ask them to record the reason for each failure. Compare a sample of their decisions. If reviewers interpret a criterion differently, improve the rubric before treating its score as reliable.
A model can help grade style, relevance, or similarity across a large test set. Treat that grade as a signal rather than the final authority. Calibrate it against human decisions, require a short reason, and send high-impact or uncertain cases to a person.
The best mix depends on the application. The principle is simple: automate what is objective, review what carries judgment, and keep the scoring process auditable.
- Establish a baseline before changing the system
Run the complete evaluation set against the current version and save the results. Record the model, prompt version, retrieval settings, available tools, and any other configuration that can affect the output.
Then change one meaningful variable at a time. Compare the new version with the baseline case by case, then review the average score. An improvement in common requests may hide a regression in a smaller but more serious group.
Keep the old failures. They are regression tests. When a production incident reveals a new failure mode, add a sanitized version of that case to the set and state what should happen next time.
- Test agents by their actions, not their prose
An agent may write a convincing explanation while choosing the wrong tool or changing the wrong record. For tool-using applications, capture the action trace as part of the evaluation.
Check whether the agent selected an allowed tool, passed the right parameters, respected read and write boundaries, asked for confirmation at the correct point, handled errors without inventing success, and stopped when the task was complete.
Evaluate side effects in a sandbox or test account. Verify the resulting state directly. A message saying "done" is not evidence that the action happened.
Include recovery cases. The agent should know what to do when a tool times out, returns partial data, or succeeds after the response is interrupted. These cases reveal duplicate actions and false success claims before they reach a live workflow.
- Create a release gate
A release gate turns the evaluation into a decision. Define the minimum pass rate for normal tasks, the maximum tolerated rate for review-required outcomes, and the failures that block release regardless of the average.
Critical safety, permission, or data-integrity failures should be visible on their own. Do not let a large number of easy passes cancel them out.
The gate should also name who can accept an exception and how long that exception lasts. Otherwise temporary workarounds become permanent, undocumented behavior.
- Use a scorecard people can inspect
A compact scorecard can include:
Version and configuration.
Evaluation set version and number of cases.
Pass, needs-review, and fail counts.
Results by task type and severity.
Critical failures and their case identifiers.
Response time and cost per completed task.
Human escalation rate.
Known limitations and the release decision.
The scorecard is useful because it connects a headline result to specific cases. Product, engineering, and domain reviewers can inspect the same evidence without reading every raw trace.
- Continue evaluating after release
Pre-release tests cover known cases. Production monitoring shows which cases the team missed. Capture user corrections, abandoned flows, repeated attempts, escalations, tool errors, and explicit feedback without storing more personal data than the workflow requires.
Review samples from production on a regular schedule. Turn important failures into new evaluation cases, fix the underlying system, and rerun the full set before release. Keep the feedback loop focused on completed tasks and serious errors rather than on whether an answer merely sounds fluent.
A practical starting plan
Choose one high-value task and write its boundaries. Create 30 to 50 distinct cases across normal, ambiguous, unsupported, and risky requests. Define pass, needs-review, and fail criteria. Add deterministic checks for structure and permissions. Have two people review a small shared sample to tighten the rubric. Run a baseline, record the configuration, and set a release gate before tuning prompts or swapping models.
The first evaluation does not need to be perfect. It needs to be repeatable, tied to real work, and honest about what has not yet been measured. That gives the team a defensible way to improve the application without relying on demos or intuition.
If you are deciding whether an LLM application is ready for production, start a conversation with a clear task, the systems it must connect to, and the risks that would block release.