How to evaluate AI agents that use tools (without fooling yourself)

Why tool-using agents are harder to evaluate than prompts#

A plain prompt has an input and an output you can score. An agent has a loop: it reads the task, chooses a tool, reads the result, chooses again, and eventually stops. Three things make that hard to evaluate.

• Non-determinism. The same task can take different paths on different runs, and some paths succeed while others fail.
• Side effects. Calling real tools can send email, create tickets or change data. You cannot casually run thousands of trials against production.
• Many valid routes. Two agents can both finish the job, one cleanly and one with ten wasteful calls or a risky shortcut.

The way to fool yourself is to test on a handful of favourite examples, run each once, see them pass and ship. The rest of this post is about avoiding that, with checks that are cheap enough to run on every change and honest enough to catch regressions.

Build the task suite from real work#

A task suite is a list of cases, each with a starting state, an instruction and a definition of success. Start small: twenty good cases beat two hundred copied from a benchmark that has nothing to do with your product.

Where to get cases:

• Real requests. Take actual tasks people gave the agent, with private data removed.
• Every bug. When a run goes wrong in production, turn it into a case before you fix it.
• Edge cases on purpose. Missing data, ambiguous requests, two matching accounts, a tool that errors, a result with a hostile instruction inside it.
• Should-refuse cases. Tasks the agent must decline or stop and ask about. Agents that never say no are dangerous.

Tag cases by category so you can see where quality changes: search, writes, multi-step, error recovery, safety. A single overall score hides the fact that you improved easy cases while breaking hard ones.

Mocked tools, recorded fixtures and real tools#

You have three options for what the agent's tools do during an eval, and a good setup uses all three for different jobs.

Mocked tools

Fake implementations that return canned data. They are fast, free and perfectly repeatable, and good for testing logic such as "does the agent paginate" or "does it recover from a rate-limit error". The danger is that your mock is more polite than reality.

Recorded fixtures

Capture real tool responses once, then replay them. You get realistic shapes, odd fields and awkward data, without network calls. Refresh the recordings on a schedule so they do not rot, and scrub personal data.

Real tools

Run against a sandbox account or a read-only connection. These runs are slow and sometimes flaky, so keep the set small, but they are the only thing that catches drift when an API changes. Never point write-capable evals at real production data.

Check the outcome first, the trajectory second#

Outcome checks ask whether the world ended up in the right state: the ticket exists with the right title, the summary mentions the three overdue invoices, no message was sent. They are robust because they allow any valid route.

Trajectory checks look at the sequence of calls. They are brittle if you demand an exact sequence, since good agents find other routes. Use them for constraints, not scripts:

• Must never call a delete tool in this case.
• Must look up the user before creating the record.
• Must use fewer than N tool calls.
• Must not repeat an identical failing call more than twice.

Prefer deterministic graders, such as comparing state, parsing fields and matching IDs, wherever you can. Use a model as judge only for fuzzy qualities like tone or completeness, give it a clear rubric, and spot-check its verdicts against human ones before you trust it.

A small eval case in TypeScript#

Here is one way to shape a case. The details will differ in your stack; the idea is that every case declares fixtures, an outcome check and trajectory constraints.

type EvalCase = {
  id: string;
  tags: string[];
  task: string;
  fixtures: Record<string, unknown>; // tool name -> recorded response
  outcome: (r: RunResult) => boolean;
  forbidden?: string[]; // tools that must not be called
  maxToolCalls: number;
};

const overdueSummary: EvalCase = {
  id: 'summarise-overdue-invoices',
  tags: ['read', 'summarise'],
  task: 'List the overdue invoices and their total.',
  fixtures: { 'invoices.list': recordedInvoices },
  outcome: (r) =>
    r.finalText.includes('3 overdue') && r.finalText.includes('4,250'),
  forbidden: ['invoices.delete', 'email.send'],
  maxToolCalls: 5,
};

function grade(c: EvalCase, r: RunResult) {
  const calls = r.toolCalls.map((t) => t.name);
  return {
    outcome: c.outcome(r),
    safe: !calls.some((n) => c.forbidden?.includes(n)),
    efficient: calls.length <= c.maxToolCalls,
  };
}

Notice the three separate verdicts. A run that gets the right answer but calls a forbidden tool should fail loudly, and a run that is right but wasteful should show up as an efficiency regression rather than a pass.

Flaky evals: run more than once and report rates#

If an agent passes a case once, you have learned very little. Run each case several times and report the pass rate, not a single result. A case that passes four out of five is a different problem from one that passes zero or five, and your reporting should make that visible.

Separate two kinds of flakiness. Model variance is real behaviour and tells you the agent is unreliable on that task, so keep it in the score. Infrastructure flakiness comes from timeouts, rate limits or a sandbox that was down, and should be retried or excluded and counted separately, otherwise your numbers become noise.

Pin everything you can: model version, prompt, tool definitions and fixtures, and record the configuration alongside every result. When a score moves, you need to know whether you changed the agent or the environment. Where your model allows it, lower the sampling temperature for evals, but still repeat runs, since lower variance is not zero variance.

Regression gates, budgets and human review#

Evals matter when they can stop a bad change. Turn them into a gate:

• Regression gate. Fail the change if the pass rate drops on any tagged category beyond a threshold you chose, or if any safety case fails at all.
• Cost budget. Track tokens and tool calls per case. A change that doubles cost for the same outcome is a regression.
• Latency budget. Measure end-to-end time per case. Agents that are right but slow lose users.

Pick thresholds as policy decisions based on what your users will tolerate, and revisit them. They are not natural constants.

Finally, read real transcripts. Each week, sample some runs, passes included, and look at them by hand. You will find graders that approve bad behaviour, cases that no longer match the product, and failure modes nobody wrote a test for. Fold what you find back into the suite.

How Setu approaches this#

Setu is a connection layer, not an eval framework, but some of its features make evaluation safer. A read-only connection lets you run realistic read-heavy cases against a real app without any risk of writes. By default, writes and deletes need approval, so a mistaken run does not silently change data. And the plain-language activity log gives you a readable record of what an agent did, which is useful for the human review sample.

Workflows can also run on schedules and webhooks with writes still waiting for approval, which makes a recurring job easy to inspect. For setup, see the quickstart, and for ready-made starting points browse the templates.

Frequently asked questions#