Human approval for AI agents: design it so people use it

Why most approval flows quietly stop working#

The failure mode is predictable. The agent asks for permission on every step, the person clicks yes forty times in a row, and by the fortieth click they are no longer reading. At that point the approval is a ritual, not a control. Worse, the first time it blocks something annoying, someone turns it off for good.

Two things follow from that. First, an approval request is a scarce resource: every unnecessary one makes the necessary ones less likely to be read. Second, the person answering is usually not watching the agent work. They get a prompt out of context, minutes or hours later, and have to decide fast.

So design approvals the way you would design an alert for an on-call engineer. Fire only when a human decision is genuinely needed, put everything needed to decide in the message, and make the answer one click.

Sort every action into read, write or delete#

The cheapest useful classification is by what the action does to the world:

• Read: fetches data and changes nothing. Listing issues, searching a calendar, reading a document.
• Write: creates or changes something, but the change is usually fixable. Posting a message, creating a ticket, editing a row.
• Delete: removes or sends something that cannot easily be taken back. Deleting a file, cancelling an event that notifies attendees, archiving a channel.

Tag each tool with its tier in the harness, not in the model's judgment. The model can be told a tool is risky, but it cannot be trusted to know. A tool definition that carries a risk field gives you one place to enforce policy.

type Risk = 'read' | 'write' | 'delete';

const tools = {
  'issues.list':   { risk: 'read' as Risk },
  'issues.create': { risk: 'write' as Risk },
  'issues.delete': { risk: 'delete' as Risk },
};

Be honest about edge cases. A read that returns private data to a third-party service, or a write that sends an email to a customer, is riskier than its tier suggests. When in doubt, classify upward.

Give each tier one of three rules: go ahead, ask, never#

Once actions have tiers, a policy is a small table: for each app and each tier, either go ahead, ask first or never. Three options are enough, and having a real never matters. Some people want an agent that can read their mail and draft replies but should be physically unable to delete anything.

Sensible defaults look like this: reads go ahead, writes ask, deletes ask. Let people loosen the policy per app as trust builds, and let them tighten it to read-only when they just want the agent to look around. A read-only connection is also the easiest thing to explain to a security reviewer.

Keep the default conservative. A person who has never configured anything should get the safe behaviour. Asking too often is annoying but recoverable; acting too freely is not.

Keep the policy outside the model entirely. The harness decides whether a call runs, waits or is refused, and the model sees only the result. If the policy lives in the prompt, a prompt injection can talk the model out of it.

Show exactly what will happen, not what the agent intends#

A prompt that says "The agent wants to update a record" is useless. The person cannot tell whether that is fine. Show the concrete action, with the real arguments, in words a non-engineer understands:

Send an email from your work account to [email protected]. Subject: "Contract draft". Body: (first lines shown). Attachment: none.

Build the preview from the actual tool call the harness is about to execute, not from text the model wrote about it. Otherwise the model can describe one thing and do another, whether by mistake or because it was manipulated. The preview and the executed call should come from the same data, byte for byte.

Good previews name the account when there are several ("your work Gmail", not just "Gmail"), show the target (which channel, which project, which file), and flag anything unusual such as many recipients or a very large change. If the action affects many items, say how many.

Approve once or approve always: scope and expiry#

The tempting shortcut is an "always allow" button. It has a place, but be careful what it covers. The scope of an approval should be as narrow as the person would expect:

• This action only: one specific call with specific arguments.
• This run: everything of this kind within the current task.
• This app and tier, from now on: a change to the standing policy, made deliberately in settings rather than in a heat-of-the-moment prompt.

Every approval should also expire. If someone approves "post this message" and the agent does not retry for three days, the context may have changed completely. A short expiry makes stale approvals harmless. When one lapses, the agent simply asks again.

Treat "always" as a policy edit, not a button on the prompt. Moving from ask to go ahead for a tier should be a visible setting the person can find and reverse later, so a single tired click does not quietly widen the agent's powers.

Retry after approval without doing the work twice#

The hard engineering part is what happens between "waiting" and "approved". The agent called a tool, the harness held it for approval, and the model cannot sit and block for hours. A workable pattern is to return a structured result immediately:

{
  "status": "approval_required",
  "message": "Waiting for the user to approve this action. Do not retry with different arguments. Try again in a few minutes."
}

The model is told to wait and call the same tool again. When the person approves, the next identical call runs. That second call is where bugs hide. If the agent changes a single argument, is it the approved action or a new one? If the first attempt partly ran, will the retry duplicate it?

Bind the approval to the call: match on the tool, the account and the arguments (or a hash of them). Make the write idempotent where the downstream API allows it, for example with an idempotency key stored alongside the approval. Then a retry, or two retries in a row, results in exactly one action.

Protect attention: fewer, better notifications#

If approvals arrive by email or push, you can easily bury the person. A few rules help:

• Rate-limit. One notification per window is plenty. If five approvals queue up in ten minutes, send one message that lists them.
• Batch in the UI. Show pending approvals in a single inbox with approve and deny buttons, so clearing five takes seconds.
• Rank. Put deletes above writes, and put anything unusual at the top.
• Reduce the source. The best fix for too many prompts is fewer prompts. If a person approves the same low-risk write forty times, that is a signal to offer a rule, not to send a fortieth email.

Track your own approve rate. If nearly everything is approved without edits, the prompts are probably too broad. If people frequently deny, the agent is proposing bad actions and the real problem is upstream.

Keep an audit trail and offer undo where it is real#

Approvals look forward; an audit trail looks back. Log every call the agent makes, with the tool, the account, who or what approved it, and the outcome. Write it in plain language ("Created issue 'Fix login bug' in your Linear workspace") so a non-engineer can scan it, and keep the raw details one click away for debugging.

Undo is the other half of safety, but only offer it where it genuinely works. Creating a draft, adding a label or posting a message you can delete are reversible. Sending an email is not, no matter what a button says. A promised undo that fails is worse than none, so mark each action reversible or not, and put a time limit on reversal so the harness is not tracking compensating actions forever.

Together, previews, approvals, logs and undo give people the thing they actually want: the ability to hand work to an agent and still know what it did.

How Setu approaches this#

Setu applies these ideas by default. New accounts ask before writes and deletes, and each app has its own rule per risk level (read, write, delete): go ahead, ask, or never. You can also make a connection read-only. When an action needs approval, the agent is told to wait and retry after the person approves; the approval covers one run and expires.

If something is waiting, Setu emails the person, at most one email every 10 minutes. Everything the agent does appears in an activity log written in plain language, and some reversible actions can be undone for 24 hours. See the concepts page for how rules work and the security overview for the wider model.

Frequently asked questions#