The most dangerous sentence in an AI project is “it worked when we tried it.” Trying an agent a few times tells you it can work. It tells you nothing about how often it fails, how it fails, or what happens on the inputs you did not think to test. Before an agent goes in front of customers or touches real data, you evaluate it — properly. Here is what that means.
Build a Golden Test Set
Start with a set of real inputs paired with known-good outputs — the questions people actually ask, with the answers you would accept. Run it on every change. This is the single highest-value thing you can do, because it turns “seems fine” into a number you can watch. When a prompt tweak or a model update quietly breaks something, the golden set catches it before your users do.
Test the Tool Calls, Not Just the Words
For an agent, the answer is only half the story. Did it call the right tool? With the right arguments? In the right order? An agent that gives a plausible response while calling the wrong API is worse than one that just gets the words wrong — it looks correct and acts incorrectly. Evaluate the actions, not only the prose.
Probe the Failure Modes on Purpose
- Hallucination and citation — are answers grounded in real sources, and can it say “I don’t know” instead of inventing?
- Prompt injection — can hostile text in a document or message hijack the agent’s instructions?
- Permissions — under pressure, does it respect access boundaries, or can it be talked into reaching data it should not?
- Edge inputs — empty, malformed, enormous, or adversarial inputs that the happy-path demo never covered
These are not exotic. They are the standard ways agents fail in production, which means they are exactly what you test before production.
Decide the Bar Before You Measure
“Good enough” is not a number until you make it one. Before launch, agree what accuracy is acceptable, what failure is unacceptable, and what the agent should do when it is unsure. An agent that is right 90% of the time needs a plan for the other 10% — usually escalation to a human. Deciding that in advance turns a vague hope into an operational standard.
Keep Measuring After Launch
Evaluation does not end at go-live. Production traces show what the agent actually did; cost and latency monitoring catch a runaway before the invoice does; regression tests catch the day a model or prompt change breaks something that used to work. The agents that stay reliable are the ones that stay measured.
This Is How You Ship With Confidence
Deploying an agent on evidence instead of optimism is the difference between an AI system your team trusts and one they quietly stop using. Software Depo builds the evaluation and observability stack into every agent we deliver, so the go/no-go decision is made on data.
Tell us what your agent must get right — and what it must never get wrong — and we will build the tests that prove it.
Pick a Scoring Method You Can Afford to Run
A golden set is only as strong as the rule that decides pass or fail. For anything longer than a single extracted value, that rule is a rubric, and the rubric has to say what makes an answer unacceptable rather than merely imperfect. An unwritten rubric produces scores that drift with whoever or whatever is applying it, and a number that drifts cannot detect a regression.
Then decide how many cases you need, because a pass rate on twenty cases is a sample, not a measurement. Twenty cases put a single failure at five percentage points, so any movement smaller than that is invisible and any movement larger than that may still be noise. Detecting a five point regression with any confidence takes on the order of several hundred cases; below that the suite can only catch gross breakage, which is worth having but should not be reported as accuracy.
Case count is limited by what each case costs to score, which makes the scoring method the decision that sets the size of the suite. The scorers available, cheapest first:
- Exact string match, or match after normalizing whitespace and case. Deterministic and nearly free, but only applicable where exactly one rendering is correct: an extracted date, a routing code, a classification label, a boolean.
- Constraint checks. Does the output parse against the declared schema, do the identifiers it returns actually exist in the system of record, are the quantities inside a possible range. These catch a large share of real failures without anyone deciding what a good answer reads like.
- End state assertions. Ignore the prose entirely and check the record the agent was supposed to create, update, or route. The strongest scorer available for any agent whose job is an action rather than an answer.
- Model grading against the written rubric, with the reference answer supplied. Handles open prose, and carries its own calibration cost because agreement with your rubric is a property of the specific grader version.
- Human review. The reference the other four are calibrated against, and the only line item on this list that does not get cheaper as the suite grows.
Record the Trajectory So Failing Runs Are Replayable
A pass or fail on the final answer tells you that a case broke, not where. The trajectory is the ordered record of what happened inside the run: each model call with the exact messages sent, each tool call with its arguments, the raw payload each tool returned, and token count and latency per step. Store it for every case in every run, not only for failures, because you need the passing traces to see what changed on the day a case starts failing.
Capture arguments as they were serialized to the tool and responses before any parsing, so a failure caused by a malformed field stays visible instead of being swallowed by a parser. Redact at capture time rather than at read time, since a trace store holding customer data inherits the access controls of the system that data came from. Log the tool schema version presented to the model alongside the trace: editing a tool description changes agent behavior with no prompt change and no code change, and it is otherwise an invisible variable.
With trajectories stored, write assertions that survive a prompt rewrite, which means asserting invariants over the run rather than the single path one passing run happened to take. The retrieval call ran before the answer that cites it. No write call ran without the read that validates its target. No tool was called with an identifier the agent was never given. No case ended with an unhandled tool error. Where the ordering is itself the requirement, assert the exact sequence and keep it strict: authorize before charge, check inventory before commit, resolve the account before writing to it. Elsewhere, an agent that re-plans, calls a tool a second time, and reaches the same verified end state has not regressed, though an assertion pinned to one exact call list will report it as one.
Maintain the Answer Key or It Will Quietly Go Wrong
Expected outputs decay. A price changes, a policy is rewritten, a product is discontinued, a document is superseded, and cases that were correct start failing for reasons that have nothing to do with the agent. That failure looks identical to a regression, and the common response is to loosen the assertion until the case passes again, which removes coverage without anyone recording that coverage was removed.
Record why each expected output is what it is, and cite the source it came from. Provenance is what stops the next engineer from correcting a case that was already right, and it is what separates a decayed key from a real defect in one step: open the cited source and check whether it still says what the key says. Cases with no provenance get deleted eventually, because nobody can defend them.
Tie re-verification to change events in the underlying sources rather than to the calendar. If the source of truth is a document store, re-verify the affected cases when a source document version changes; if it is a database, when the schema or the reference data behind those cases moves. A usable case record carries:
- The input in the exact form the agent receives it, including any prior conversation turns and system context that arrive with it.
- The expected output or the rubric.
- The tools the agent was permitted to call for that case, because the same input with a wider tool surface is a different test.
- Pinned identifiers and versions for every source document or record the correct answer depends on.
- Who verified the answer and when, so a stale key is visible as stale instead of assumed correct.
Attribute Each Failure to a Layer Before Changing Anything
An aggregate pass rate tells you how often, never why. Label every failing case with the layer that actually broke: retrieval returned the wrong context, the model chose the wrong tool, the tool was right but the arguments were wrong, the tool returned an error the agent ignored and answered anyway, or the context was correct and the synthesis was not. Those five failures have five different fixes, and only one of them lives in the prompt.
The diagnostics separate them cheaply once trajectories are stored. Replace the retrieved context with context you know contains the answer and re-run: if the case passes, the fault was retrieval, not reasoning. Force the correct tool choice and re-run: if it passes, the fault was selection. Send the exact recorded arguments to the API by hand: if the API rejects them the fault was argument construction, and if the API returns the right data while the agent still answered wrong, the fault was synthesis. Before any of that, run the failing case ten times unchanged, because a case that fails two times in ten is a variance problem and needs a different response than one that fails ten times in ten.
Label first because prompt edits are global. A change written to fix one synthesis failure moves every case in the suite, including the ones that were passing, and without layer labels you cannot tell whether the next run improved or simply traded one failure for another. Retrieval faults are fixed in chunking, indexing, and filters. Selection faults are more often fixed by cutting the number of tools the agent is offered than by describing them better, since selection accuracy degrades as the tool list grows and as tool descriptions overlap.
Every Score Belongs to a Configuration
A number without the setup that produced it cannot be compared against anything, so record the full version set alongside every evaluation run: the model identifier including its dated snapshot, the decoding parameters, a hash of the prompt template, the tool schema version, the retrieval index build, and the revision of the golden set itself.
Pin the dated snapshot rather than a floating alias. Aliases that resolve to the newest release move on the provider's schedule, not yours, so a result that looks like a prompt regression can be a model change nobody on the team made, and the two are indistinguishable after the fact. When you do move versions deliberately, run both against the same golden set revision before switching, so the delta you attribute to the new model is measured rather than assumed.
Expect a spread even with everything pinned. Greedy decoding reduces run-to-run variation but does not eliminate it, since batching, hardware, and provider-side infrastructure changes all affect results, and any tool that returns live data adds variation of its own. Run the full suite several times against an unchanged configuration at the start to learn its normal spread, then treat only deltas larger than that spread as signal. Without that baseline a team chases noise, and after a few false alarms it stops reading the results at all.
Frequently Asked Questions
What is a golden test set, and what is it not?
It is a fixed, versioned collection of real inputs paired with outputs someone has verified as acceptable, re-run on every change. It is not a benchmark: the scores mean something against your own previous runs on the same set and nothing against another team's number or a public leaderboard. It is also not coverage of production, because it contains only the inputs somebody thought to include, which is why it runs alongside production traces rather than instead of them.
How do you choose which cases go into the set?
Stratify instead of sampling uniformly. Bucket by intent, by the tool path a correct run should take, by data source, and by difficulty, then make sure each bucket holds enough cases to fail visibly on its own, because uniform sampling from live traffic buries the rare and expensive paths under the common easy ones. Grow the set by promoting every diagnosed production failure into a permanent case, since that is the only reliable supply of inputs nobody thought to test. Keep the buckets separate in reporting as well, because an overall pass rate can hold steady while one bucket collapses.
What has to already exist before evaluation can start?
A runnable instance of the agent outside production with the same tool surface it will have in production, credentials or test instances for every system it calls, and a set of inputs you are permitted to use. Real inputs often carry customer data, so either they are redacted before entering the test set or there is a basis for using them as they are. You also need a person who can state authoritatively what the correct answer is; without that you have inputs and opinions rather than a golden set.
How do you evaluate an agent that writes data instead of just answering?
Give it a sandbox seeded with known fixtures, reset it between cases, and assert on the end state of the records rather than on the wording of the reply. Where a system has no usable sandbox, record the real API responses once and replay them, which makes the suite deterministic and cheap enough to run often. The tradeoff is that recorded responses go stale, so pair the replay suite with a small contract test that calls the live API on a schedule and fails when the real responses stop matching the recordings.
How does this fit into a build and release pipeline?
Instrument for traces first, because a failing case in CI with no stored trajectory gives you nothing to debug. Then assemble the verified set, run it against a pinned configuration, write results and traces to a store keyed by run ID, and only once those exist define which thresholds gate a release. Run a fast subset per commit and the full suite pre-release or on a schedule, since several hundred agent runs cost real money and real minutes. Any non-deterministic check also needs a stated retry and threshold policy up front, or the first flaky failure teaches the team to ignore the gate.
Can a model grade the agent's output?
Yes, within limits. Give the grader the rubric and the reference answer and ask for a judgment against them rather than a general opinion on quality, and run it as a separate call that never sees the agent's own reasoning. Validate the grader against human labels on a held out subset and re-validate whenever its model version changes, because agreement with your rubric belongs to that specific version. Test for the known biases directly: graders tend to reward longer and more confident answers, so include cases where the correct output is short, a refusal, or an admission that the agent cannot answer.
What does the customer have to decide rather than the engineering team?
Who owns the answer key. When two reviewers disagree about what an acceptable output is, one named person's judgment has to be authoritative, and that person sits on the business side rather than the build team. Whether the agent may take write actions in production, and precisely which ones, is the same kind of decision. So is which thresholds block a release and which only get reported, since engineering can measure any threshold but cannot decide which failures the business is willing to absorb.
