The most dangerous sentence in an AI project is “it worked when we tried it.” Trying an agent a few times tells you it can work. It tells you nothing about how often it fails, how it fails, or what happens on the inputs you did not think to test. Before an agent goes in front of customers or touches real data, you evaluate it — properly. Here is what that means.
Build a Golden Test Set
Start with a set of real inputs paired with known-good outputs — the questions people actually ask, with the answers you would accept. Run it on every change. This is the single highest-value thing you can do, because it turns “seems fine” into a number you can watch. When a prompt tweak or a model update quietly breaks something, the golden set catches it before your users do.
Test the Tool Calls, Not Just the Words
For an agent, the answer is only half the story. Did it call the right tool? With the right arguments? In the right order? An agent that gives a plausible response while calling the wrong API is worse than one that just gets the words wrong — it looks correct and acts incorrectly. Evaluate the actions, not only the prose.
Probe the Failure Modes on Purpose
- Hallucination and citation — are answers grounded in real sources, and can it say “I don’t know” instead of inventing?
- Prompt injection — can hostile text in a document or message hijack the agent’s instructions?
- Permissions — under pressure, does it respect access boundaries, or can it be talked into reaching data it should not?
- Edge inputs — empty, malformed, enormous, or adversarial inputs that the happy-path demo never covered
These are not exotic. They are the standard ways agents fail in production, which means they are exactly what you test before production.
Decide the Bar Before You Measure
“Good enough” is not a number until you make it one. Before launch, agree what accuracy is acceptable, what failure is unacceptable, and what the agent should do when it is unsure. An agent that is right 90% of the time needs a plan for the other 10% — usually escalation to a human. Deciding that in advance turns a vague hope into an operational standard.
Keep Measuring After Launch
Evaluation does not end at go-live. Production traces show what the agent actually did; cost and latency monitoring catch a runaway before the invoice does; regression tests catch the day a model or prompt change breaks something that used to work. The agents that stay reliable are the ones that stay measured.
This Is How You Ship With Confidence
Deploying an agent on evidence instead of optimism is the difference between an AI system your team trusts and one they quietly stop using. Software Depo builds the evaluation and observability stack into every agent we deliver, so the go/no-go decision is made on data.
Tell us what your agent must get right — and what it must never get wrong — and we will build the tests that prove it.
