BYD TechThe short answer
To evaluate an AI agent or LLM feature, first write down what success means in measurable terms, then build a set of real example inputs with the results you expect. Run the feature on that set after every change and grade the outputs, with code checks where answers are exact, a model grader for judgment calls, and people reviewing a sample. After launch, keep logging real conversations and add the failures to the test set.
Key points
- Define success before building: specific, measurable, tied to the task.
- A test set of real inputs is worth more than any demo.
- Grade with code where possible, a model where needed, people for a sample.
- Rerun the evals on every prompt, model or tool change.
- Real failures from production become new test cases.
Why demos are not enough
AI features look good in a demo because the demo uses the inputs the builder already tried. Real users bring different wording, missing information and edge cases. Model output also varies between runs, and a small prompt change can fix one case and break three others. Evals, short for evaluations, are the test suite that catches this.
Step 1: Define success
Anthropic’s guidance is to make success criteria specific, measurable, achievable and relevant. "Good answers" is not a criterion. "Answers the policy question correctly, cites the right document, and refuses when the answer is not in the documents" is.
| Dimension | Example measure |
|---|---|
| Correctness | Share of answers that match the expected answer |
| Grounding | Share of claims supported by the retrieved sources |
| Task completion | Share of agent runs that finish the job correctly |
| Safety | Share of out-of-scope requests correctly refused |
| Cost and speed | Tokens and seconds per request |
Step 2: Build a test set
- Collect real inputs: support tickets, search queries, documents, past requests.
- Write the expected result or the rules a good result must meet.
- Include hard cases: vague questions, missing data, requests the feature should refuse.
- Start with dozens of cases, and grow it as you find failures.
- Keep the set in your repository, next to the code it tests.
Step 3: Grade the outputs
| Grader | Good for | Watch out for |
|---|---|---|
| Code checks | Exact answers, formats, required fields, tool calls | Cannot judge tone or reasoning |
| Model as grader | Judgment: relevance, grounding, following rules | Needs a clear rubric, and checking against people |
| Human review | The final word on quality | Slow, so use it on samples |
For agents, grade the path as well as the result: did it call the right tools, in a sensible order, without unsafe actions?
Step 4: Keep evaluating after launch
- Log inputs, outputs and tool calls, with care for personal data.
- Let users flag bad answers, and review flagged ones every week or so.
- Turn each real failure into a test case.
- Rerun the full set before changing the prompt, the model or a tool.
Many failures come from invented answers. See How to reduce AI hallucinations. Evals belong in the plan from the start of any AI agent development project.
Sources
Checked when this guide was written.
