AI evaluations, guardrails, and observability are what separate an AI demo from an AI system you can put in front of customers. Software Depo builds the testing and monitoring that let you deploy an agent with evidence — measured behavior before rollout, and visibility into what it does in production.
You Cannot Ship What You Cannot Measure
“It seemed to work when we tried it” is not a deployment standard. Before an agent goes live, we measure it against golden test sets, check its tool calls, and probe the ways it could fail — so you make the go/no-go decision on data, not vibes. This is one of our strongest areas: our agents ship with the evaluation stack built in.
What We Test
- Golden test sets — known inputs with known-good outputs, run on every change
- Tool-call testing — does the agent call the right tool with the right arguments?
- Hallucination and citation testing — are answers grounded and verifiable?
- Prompt-injection resistance — can hostile input hijack the agent?
- Permission testing — does it respect access boundaries under pressure?
- Regression testing — did a prompt or model change quietly break something?
Observability in Production
- Production traces — see exactly what the agent did and why
- Cost monitoring — catch a runaway token spend before the invoice does
- Latency monitoring — know when responses are slowing
- Human escalation — a clean path for the cases the agent should not handle alone
Independent Review
For security-critical deployments, evaluation pairs well with independent assessment. Our sister practice BulletproofSoft reviews AI and MCP deployments from a security angle — a different team, by design.
Tell us what the agent must get right — and what would be unacceptable for it to get wrong. We will build the evaluations and monitoring that prove it before and after launch.
Related services
- Agentic AI Consulting and Implementation — Deciding what to build and how to prove it: agentic AI consulting.
Evaluations Run Before Deploy, Guardrails Run at Request Time
The two halves of this work happen at different moments and are easy to conflate. Evaluations are offline: a fixed set of cases, run against a candidate version of the agent, producing a result someone reads before anything is promoted. Guardrails are online: code sitting in the request path that can refuse, rewrite, or reroute a single live interaction no test anticipated. Even a thorough evaluation suite does not remove the need for guardrails, because the inputs nobody thought of are precisely the ones that arrive in production.
Every guardrail is a tax on every request, in latency and in tokens, and they accumulate quietly. Each one has to earn its place against a specific failure you decided you cannot absorb. In practice the useful ones are narrow: a check on the shape and scope of a tool call before it executes, a check that an answer carries a citation when policy requires one, and a hard stop before the agent writes to a system of record with arguments outside an approved shape. Those are functional controls in the request path, distinct from the independent security assessment of the same agent and its connectors, which by design sits with a different team.
- An evaluation failure is a conversation before release. A guardrail firing is an event in production, and it needs a destination and a reader.
- Guardrails that only log are not guardrails. If a check cannot block, refuse, or reroute, it is instrumentation, and calling it a control is how a gap gets missed.
- A guardrail added after an incident, with no matching case added to the suite, means the same fix has to be rediscovered the next time the prompt changes.
How the Suite Is Built, and What Keeps It True
The suite is built backwards from failure rather than forwards from features. Failure modes get enumerated first, in your language and about your systems: the wrong record updated, an answer asserted with nothing retrieved behind it, one account's request answered with another account's context in scope. Each mode becomes a class of cases, and each class needs a grader that decides pass or fail without a person reading every run. A suite that requires human review of every result does not survive its first deadline.
Graders are where these programs quietly succeed or fail. A loose grader passes broken behavior. A strict one fails on wording, gets muted, and takes the credibility of the rest of the suite with it. We write graders against the property that matters rather than the exact string, prefer deterministic checks wherever a property can be expressed as one, and keep model-graded checks for the parts that are genuinely subjective, with the grading prompt under version control like any other code.
- Cases come from real transcripts and real records where you can share them, and from synthesized inputs shaped like the real ones where you cannot.
- Model versions are pinned, so a provider-side change surfaces as a test failure rather than as a mystery in production.
- Every production failure becomes a case, and the case stays after the fix ships.
- Flaky cases are quarantined and repaired, not deleted. A suite people do not trust is a suite people route around.
- Disagreements between a grader and a human reviewer are recorded as their own data, because that is the signal that a grader needs correcting.
What the Work Depends On From You
One input cannot come from us: the ranking. Not only which failures matter, but which one you would accept first if forced to choose between two, and whose name sits on that order. The ranking is what decides which failing cases stop a promotion and which only file a ticket. Without it, every failure becomes an argument held at the worst possible moment, usually the night before a launch.
The rest is access and adjudication. We need a way to drive the agent outside a chat window, credentials scoped to an environment where tool calls can run without touching live systems, and one internal person who can look at a disputed case, say whether the answer was right, and be treated as final. That role is small in hours and decisive in outcome. If nobody can settle what correct means, the graders end up encoding our guess about your business.
- Representative inputs: transcripts, tickets, documents, records. Redacted is fine. Invented from scratch is not.
- A test environment, or a clearly marked set of records that can be written to and reset.
- Named owners for the connectors and MCP servers the agent calls, so a failing tool call routes to the team that owns the tool instead of getting blamed on the model.
- A decision about what a failing suite is allowed to stop. That is a business call, and it holds only if you are the one who makes it.
When a Standalone Evaluation Engagement Is Not the Right Call
This section is about standalone engagements: evaluation and observability work on an agent or workflow you already run, built by another vendor or by an earlier version of your own team. Agents we build carry the evaluation stack with them, so the question of whether to add one does not arise there. For work you already have running, it is a fair question, and the answer is sometimes no.
None of these are permanent verdicts. The deterministic case stays deterministic, but the other three usually change: the tool surface settles, human review gets lifted for the routine cases, or someone takes ownership. Saying no early is not a lack of appetite for the work. It is what makes the work produce a maintained artifact instead of a snapshot that ages out.
- The automation is deterministic. A scripted integration or a rules engine needs conventional software tests. Adding graders to code that never calls a model measures nothing.
- The agent is still changing shape week to week. Formalizing a suite around an interface you are about to replace produces cases you throw away. A short list run by hand is more honest until the tool surface settles.
- A person approves every action before it takes effect. That reviewer is the control. Instrument what the agent did and what it cost, and spend the evaluation effort on workflows that act without review.
- Nobody will own it. An unowned suite degrades into a red build everyone learns to ignore, which is worse than no suite, because from the outside it looks like coverage.
Frequently Asked Questions
What is an AI evaluation suite as a deliverable, and what is it not?
It is a repository: the cases, the graders that score them, the harness that runs both, and the configuration that decides which failures stop a release. It lives in your version control next to the agent, and it is triggered by the changes that can move behavior, which means a prompt edit, a model version change, a tool definition change, or a new or updated MCP server or connector. It is not a benchmark score and it is not a certification. It does not say the agent is safe in general; it says this version of this agent, on these cases, behaved this way on this date.
What has to already exist before you can build one?
A working agent, or a prototype far enough along that its tool surface is real, plus a way to drive it programmatically rather than by typing into a chat window. Concretely that means an API or entry point the harness can call, credentials scoped to a test environment, and tool or MCP server endpoints that can be pointed at data you are willing to have written to. If the agent exists only as a hosted assistant with no programmatic entry point, that gap gets closed first, because a suite that cannot run unattended does not get run.
How does the rollout actually proceed, in order?
Failure modes and their ranking come first, because they decide what is worth testing at all. Then the harness and a small set of cases go in and run against the agent as it exists today, which produces a baseline: not a target, just an honest record of current behavior. Cases expand from there until the failure classes you named are covered, and only then do suite results get wired into whatever gate promotes a change. Production instrumentation goes in last, in the order the monitoring section above describes.
What do we have to provide, and what has to be our decision?
You provide the environment, the credentials, representative inputs, and time from someone internal who can settle disputed answers. We provide the harness, the graders, and the cases. Two decisions stay yours: the ranking of failures described above, and the threshold question of which failing cases block a promotion versus which only raise a ticket. We will recommend both, but a threshold we set alone is the first thing overridden the moment it becomes inconvenient.
How does this integrate with the systems we already run?
The suite runs as a job in the CI system you already use and fails a build the same way your existing tests do, so no new tool has to be adopted to see a result. What the running agent emits is written in a form your current observability platform can ingest, so the output lands beside your other service telemetry instead of in a separate dashboard nobody opens. The harness calls the same tool and MCP server interfaces the agent uses in production, pointed at test endpoints, which makes integration a configuration exercise rather than a parallel implementation. If you have no CI and no observability platform, that is a prerequisite to settle before the suite, not something to build around.
Where does our data live, and who has access?
Cases, transcripts, traces, and grader output live in your infrastructure: your repository, your observability platform, your accounts. We work under credentials you issue and can revoke, scoped to the environments you name, and access ends when you end it. Where a realistic case needs real records, redacted records or synthetic stand-ins built from the shape of the real data are usually sufficient, and that call is yours case by case. Model-graded checks do send case content to whichever model provider you have already selected for grading, which is worth deciding deliberately rather than discovering later.
What does an evaluation suite not cover?
It does not repair the sources the agent reads from. If the documents behind a retrieval step are stale or wrong, citation checks will report grounded answers grounded in bad material, and that is a content problem upstream of any suite. It also cannot evaluate a closed vendor product you have no way to drive or instrument. And it does not settle policy: a suite can demonstrate that the agent refuses a category of request, but deciding which categories must be refused is your call, not a testing question.