How to Build Evals for AI Agents — A Practitioner's Guide

Before deploying an AI agent in your organisation, one question will end the conversation if you can't answer it: what is the hallucination rate? This guide explains what evals are, why they matter, and how to build them from scratch.

The question that ends the meeting

Imagine you have built an email triage agent. It reads incoming messages, categorises them, drafts responses, and routes exceptions to a human. It works well in testing. You want to roll it out across the organisation.

You present it to the CIO, or the CFO, or the head of risk. They ask: "What's the error rate?"

If you can't answer with a number — not a vague impression, an actual quantified figure — the conversation ends there. As Aki Ranin, AI transformation advisor and keynote speaker at apidays Singapore 2026, puts it: "If you can't tell me what the percentage risk of hallucinations is, it's a complete no-go. The conversation stops right there."

That number comes from an eval. And without an eval, you are, in his words, "running in the dark."

What is an eval?

An eval (short for evaluation) is a structured set of test cases used to measure an AI model or agent's performance on a specific task. Think of it as the AI equivalent of a software test suite — but instead of checking whether code produces the correct output for a given input, an eval checks whether an AI agent produces responses that meet defined criteria of quality, accuracy, and reliability.

An eval has three essential components:

1. A set of representative inputs. These are prompts, queries, or task instructions that reflect the real range of situations the agent will encounter in production. A good eval set is not cherry-picked to make the agent look good — it includes edge cases, ambiguous inputs, and examples that are likely to cause failure.

2. Expected or acceptable outputs. For each input, you need a reference against which the agent's actual output can be judged. This might be an exact correct answer (for factual retrieval tasks), a rubric-based score (for tasks where quality is subjective), or a reference output from a stronger model acting as a judge.

3. A scoring method. The mechanism that converts the comparison between actual and expected outputs into a consistent, repeatable metric — a number you can track over time, compare across models, and present to a CFO.

Why evals are the prerequisite for everything else

The practitioner case for evals goes well beyond governance and risk management. Evals are the enabling technology for cost optimisation — and the cost argument, in 2026, is urgent.

Aki Ranin described the Shopify approach at apidays Singapore 2026. Shopify had a single, relatively narrow agentic use case running on a frontier model. The bill was approximately $5.5 million per year. By building a rigorous eval suite and using it to validate progressively smaller, cheaper models, they stepped down to a 70-billion parameter open model. Cost: approximately $70,000 per year. A reduction of over 98%, with performance maintained or improved.

The mechanism is simple: you start with the best model available, confirm it achieves acceptable performance on your eval suite, then move to the next smaller model and test again. You keep stepping down until performance drops below your threshold. The cheapest acceptable model is your production model.

This process is only possible if you have evals. Without them, you have no basis for the comparison, no confidence that the smaller model is performing adequately, and no defensible number to present when someone asks why you are not using the most capable model available.

As Manjunath Bhat, Distinguished VP Analyst at Gartner and keynote speaker at the same conference, made the point directly: "If you're not evaluating your models, your agents, then there's nothing for you to optimise." He went further, arguing that eval-driven development — defining success metrics before building the agent, then building toward them — is the AI-era successor to test-driven development. "In the traditional software engineering days, we valued behaviour-driven development, test-driven development. In the agentic era, that becomes eval-driven development."

Step 1: Define the use case precisely

Before you write a single test case, you need a clear, bounded definition of what the agent is supposed to do. This is harder than it sounds. "Summarise emails" is not a use case definition. "Classify incoming customer service emails into one of seven predefined categories and extract the customer account number when present" is.

The more precisely you define the task, the more tractable your eval becomes. Vague tasks produce vague evals, which produce meaningless metrics.

For each use case, define:

• What goes in (the input format and range)
• What comes out (the expected output format)
• What counts as correct (the success criteria)
• What counts as unacceptable (the failure modes, especially those with real consequences)

The failure modes are particularly important in regulated environments. In the legal context described by Shin Wee Chuang of Pand.ai — whose team built an on-premises AI system for law firm Allen & Gledhill — the critical failure mode was not producing a mediocre summary; it was generating legally incorrect content that a lawyer might rely on. Your eval needs to specifically test for the failure modes that matter in your context.

Step 2: Build your eval dataset

The eval dataset is the collection of (input, expected output) pairs you will use to measure performance. Building it well is the most labour-intensive part of the process — and the part most often skipped.

Start with real examples. The best eval cases come from your actual use case: real emails, real documents, real queries. Synthetic examples generated by an LLM are faster to produce but may not reflect the messiness and edge cases of real-world data. Use them to supplement, not replace, real examples.

Include edge cases deliberately. A dataset of easy cases produces an artificially high accuracy score. Include ambiguous inputs, unusual formats, borderline cases, and the kinds of inputs most likely to cause the agent to hallucinate or fail. If you know your agent struggles with multi-part questions, include multi-part questions.

Annotate with domain experts. For tasks where correctness requires domain knowledge — legal document review, medical summarisation, financial analysis — the expected outputs need to be defined by people who actually know the domain. This is what Aki Ranin described from the Shopify case study: "We have a lot of human-annotated data, which allows them to be confident on the performance and accuracy of these kind of agents." There is no shortcut here. Human annotation is expensive and slow, but it is the foundation of a trustworthy eval.

Aim for breadth before depth. A small dataset of 50 carefully chosen, well-annotated cases is more useful than 500 cases that all look similar. Cover the distribution of real inputs, not just the common case.

Step 3: Choose your scoring approach

How you measure whether the agent's output is correct depends on the nature of the task. There is no universal method — you need to match the scoring approach to the output type.

Exact match works for tasks with unambiguous correct answers: extracting a date, classifying into a fixed category, retrieving a specific account number. If the agent says "Category B" and the correct answer is "Category B", that's a pass. Simple to implement and interpret.

Rubric-based scoring works for tasks where quality exists on a spectrum: summarisation quality, tone of a drafted response, completeness of an analysis. You define a rubric — a set of criteria with descriptions of what constitutes a 1, 3, or 5 on each dimension — and score outputs against it. This requires human judgment or a strong proxy for it.

LLM-as-a-judge uses a capable language model to score your agent's outputs against the reference answers or the rubric. This scales much better than human scoring and can produce consistent results on well-defined rubrics. It introduces its own error rate (the judge model is not perfect), but for many use cases the efficiency gain justifies the tradeoff. Aki Ranin notes you can use this approach, but cautions: "There's no real way around you having to probably annotate a bunch of data" as the ground truth foundation.

Traditional ML metrics such as F1 score — a metric that balances precision and recall, familiar from machine learning — apply when the task involves classification, information retrieval, or other structured prediction. These are particularly useful when you have a large labelled dataset and want a single headline metric. Aki Ranin referenced F1 directly from the Shopify case study as the metric they used to compare model performance across the cost optimisation process.

DSPY is a prompt optimisation framework worth knowing about in this context. It provides tools for quantifying agent performance on specific tasks and optimising prompts systematically — essentially automating part of the process of finding the best configuration for your eval suite. The Shopify team used it as one of the tools in their cost reduction process.

Step 4: Run the eval and establish your baseline

Once you have your dataset and scoring method, run your current agent (or the frontier model you are starting with) against the full eval set and record the results. This baseline is your reference point for everything that follows.

A baseline serves several purposes:

• It tells you whether the task is even feasible. If the best available model only achieves 60% accuracy on your eval set, the problem may be in the task definition, the eval cases, or the fundamental difficulty of the task — not just the model choice.
• It gives you a threshold. Decide, before you start optimising, what accuracy level is acceptable for production. This is a business decision, not a technical one — it depends on the stakes, the regulatory environment, and what human oversight remains in the loop.
• It enables comparison. Every subsequent experiment — different model, different prompt, different retrieval strategy — produces a result you can compare directly to the baseline.

Document everything: the model version, the prompt, the temperature settings, the retrieval configuration. Evals are only useful if the results are reproducible.

Step 5: Optimise using the eval as your compass

With a baseline established, you can begin the model-stepping process Aki Ranin described. The logic is:

1. Start with the frontier model. Confirm it meets your performance threshold on the eval.
2. Step down to the next tier — a smaller or cheaper model.
3. Run the eval. Does performance still meet the threshold?
4. If yes, step down again. If no, step back up — or investigate whether prompt optimisation, better retrieval, or other configuration changes can close the gap.
5. The cheapest model that meets the threshold becomes your production choice.

This is not purely a cost exercise. Smaller, task-specific models often produce more consistent, auditable outputs than large general-purpose frontier models. As Aki Ranin observed, a cascade of specialised agents — each doing one thing well — "gives you just more transparency around and trustworthiness with the AI" than a single large model trying to do everything.

The optimisation does not stop at model selection. Once you know which model you are using, you can use the eval to:

• Tune the system prompt for this specific model (prompts that work well with one model do not always transfer)
• Set appropriate max_tokens limits — once you know what valid outputs look like from your eval set, you can constrain the agent's output window appropriately
• Identify specific failure patterns to address through better retrieval, additional guardrails, or human escalation paths

Step 6: Build evals into your ongoing process

A one-time eval at deployment is necessary but not sufficient. Models change (providers update them without notice), data distributions shift, and new edge cases emerge that your original eval set did not cover. Evals need to become part of your operational process, not a pre-launch checkbox.

Manjunath Bhat at Gartner describes this as the "agent development lifecycle" (ADLC) — a formal methodology for building, evaluating, and iterating on agents that mirrors the discipline of traditional software development. "Agents are never going to be right the very first time we build them," he said. "In the new agentic era, we will have a definition of good enough — and with feedback loops, you're constantly getting feedback from your test environments, from your staging environments, from production."

Practically, this means:

• Running your eval suite automatically on a schedule (weekly, or whenever the underlying model is updated)
• Monitoring for performance drift — a drop in eval scores is an early warning that something has changed in the production environment
• Expanding the eval dataset over time as you encounter new failure modes in production
• Logging production inputs and outputs so that real failures can be converted into new eval cases

The tooling landscape

You do not need to build eval infrastructure from scratch. A growing market of dedicated platforms has emerged specifically for AI evaluation and observability. Manjunath Bhat surveyed this landscape at apidays Singapore 2026, noting vendors including Braintrust, DeepChecks, Evidently AI, Fiddler AI, Galileo (recently acquired by Cisco), and LangFuse (an open-source option, acquired by ClickHouse). He described it as a nascent but rapidly consolidating space — "almost every other week you have platforms getting acquired left and right."

For most organisations starting out, the tool choice matters less than the discipline of building and maintaining a real eval dataset with genuine human-annotated ground truth. A spreadsheet of 100 well-chosen, well-labelled test cases and a simple scoring script will tell you more than a sophisticated platform running against a weak dataset.

What your eval cannot do

An eval measures whether your agent performs well on the cases in your eval set. It does not guarantee performance on cases outside that set, and it does not eliminate risk — it quantifies and manages it.

The goal is not 100% accuracy. It is an acceptable, known, defensible error rate — one that you have agreed with your stakeholders is tolerable given the use case, the stakes, and the human oversight in place. Aki Ranin is direct about this: "It's never going to be 0% hallucination risk. It's never going to be 100% accuracy. Depending on the use case, how critical it is, whether you're subject to regulations — you're going to have to make that determination."

That determination, made with real data from a real eval, is how you move from an AI prototype to a production system that an organisation can actually trust.

Further reading

• Shopify's cost optimisation case study (search: "Shopify agentic AI case study 2026") — the source of the $5.5M → $70K example
• DSPY documentation: dspy.ai — prompt optimisation framework referenced in this guide
• LangFuse: langfuse.com — open-source eval and observability platform
• Braintrust: braintrust.dev — eval platform with strong developer tooling

This article was produced for The Loop Asia, drawing on presentations and conversations from apidays Singapore 2026. Contributing sources: Aki Ranin (AI [r]ecursive), Manjunath Bhat (Gartner), Shin Wee Chuang (Pand.ai).

The Loop Asia: www.theloop.asia