Where do you draw the line when agents can call real tools?

for public stuff, the approval needs the actual account + audience + final text. otherwise yes turns into a rubber stamp pretty fast. let agents research and draft freely, but make posting a one-shot action bound to that exact text.

Checklist first, but not quite the one you just wrote down.

The reason is retrofit cost, and it splits your list cleanly. A broker is a refactor: you can put one in front of a running system later and route existing calls through it, and nothing you already did is lost. A missing record is not a refactor, it is permanent. Every call that ran before you had a schema is unrecoverable, and the field you will most want back is the one nobody logs on day one, which is the reason the agent gave for the call. Reading a log cold months later, that field is what separates a misconfiguration from an adversarial retry.

The one item that genuinely has to be day one is identity, for the same reason: a log with no subject cannot be queried later, and you cannot backfill who an old call was acting for. Buckets, rollback, approvals and the gateway can all arrive later without invalidating what you already stored.

The item I would move rather than keep is cost. A spend limit is not a per call decision, and a checklist evaluated per call will pass every single call while the loop burns, because each one is individually reasonable. Budget is a property of the job, which is the same key as your last line about tracing every call back to one job. If the run ID carries the budget, the cost check happens where the loop is visible instead of where it is invisible.

Shortest honest version: name the subject and store the stated reason from day one, enforce wherever is cheapest today, and move enforcement into a broker at the point where the same enforcement logic starts appearing in two places.

This is much more useful than I expected. The pattern I am hearing is that read/write alone is too shallow.

A better checklist seems to be: who is the agent acting for, what bucket is this action in, can it be rolled back, what state was the approval based on, what did the run cost, and can every call be traced back to one job/reason.

The two additions I had not weighted enough are cost limits and provenance from untrusted reads. A safe read can still change the next action or burn money in a loop.

For people running this in production: would you rather start with a lightweight checklist/log schema first, or is a gateway/broker necessary from day one?

Interesting question. I think the challenge is not only what agents can do, but how users understand and trust the actions they take.

Memory, transparency and control might become important parts of future AI products.

For me the line is reversibility. Reading files, running tests, even opening a PR are fine to hand off fully because I can review before anything ships. The moment an agent can send an email, push to prod, or move money, I want a human confirmation step in the loop no matter how good its track record has been, because the one time it goes wrong is usually the one time nobody was watching closely. I have found it easier to draw the line by reversibility than by trying to rate how trustworthy a given agent is that week.

One axis the blast-radius buckets miss (and it bit us in production): cost. Our worst agent incident wasn't a dangerous write - every call was in the "safe read" bucket - it was a pathological input that made the pipeline loop cheap reads and model calls until the job cost ~20x its estimate. Reversibility was fine; the money was already burned.

So next to the read/write/irreversible taxonomy we run a hard per-job budget: every tool call carries a cost estimate, the gateway keeps a running total per job, and crossing the ceiling kills the run mid-flight - same no-exceptions rule as the approval gate. Unexpected bonus: it's the cheapest early-warning signal we have. Jobs that hit the budget ceiling are almost always misbehaving in some other way too (bad input, loop, wrong plan), so the cost guard catches logic bugs the taxonomy never would.

So for your checklist: four lines, not a tree. Can it be rolled back, who approved it, what did it cost, and can I trace every call to one job id. If the infra answers those, the rest is policy.

The sharpest thing in this thread is a caveat nobody picked up: thevarun’s point that with tool-calling, a read is how untrusted text gets into the context that chooses the next call. Six comments here classify reads as safe, and in isolation they are. The trouble is that the bucket taxonomies being described, including the good ones, classify calls independently and statically. The real unit of risk is the sequence, and a read with zero blast radius can still change which action gets selected next.

Disclosure, I work at Ojin and we build real-time conversational agents, so this is a daily problem for us rather than a hypothetical.

The thing that makes the bucket model actually hold is taint tracking. Mark any content that entered context from an untrusted read, propagate the mark, and surface it at the gate. Your approval prompt then stops being "the agent wants to send an email" and becomes "the agent wants to send an email, and the recipient address came from text it read in a support ticket." That provenance is the whole difference between a human approving meaningfully and rubber-stamping, which is the reflex-approval trap MchineArenaDev described from the other direction. A gate with no provenance trains the behaviour it exists to prevent.

One addition to your checklist, as the first branch rather than a tuning knob: can your product tolerate a human pause at all. Everything above assumes latency tolerance. In a live voice conversation there is none, because a human-in-the-loop gate is a two second silence and the interaction is already broken. When that is the situation, per-call approval is simply unavailable, and the entire budget moves to capability restriction at session start. The agent gets a narrow toolset scoped before the conversation begins and there is no runtime escalation path. More restrictive, and the only version that works when you cannot stop and ask.

So the checklist I would write branches on two questions before it reaches any per-tool logic. Can we pause and ask a human. And is this argument derived from something the agent read, rather than from something the user or the system supplied. The first tells you whether approvals are even an option, the second tells you what the approval has to show.

The line we draw isn't really about the tools, it's about the blast radius of each call. We sort every tool an agent can reach into three buckets:

Read-only (fetch order status, look up a record): agent calls freely, no approval, full logging.

Reversible writes (draft a reply, create a ticket, tag a record): agent acts on its own, but everything is logged and a human can undo it.

Irreversible or external-impact (move money, message a customer, delete anything, hit a paid API): hard approval gate, every time, no exceptions.

The test we give clients is one question: if the agent gets this wrong, can we roll it back before anyone notices? If yes, let it run. If no, gate it. That replaces most of a decision tree.

On your other three points, they're separate layers and worth not collapsing into the agent. The thing that solves all of them is a gateway sitting between the agent and the real tools. The agent never holds raw keys, it asks the gateway to "send email" and never sees the key, which also means you revoke access at the gateway without rotating keys everywhere. Identity rides as context on each call, so the agent is stateless about who it's acting for and the gateway enforces that it can only touch that customer's scope. And you log at the gateway, not the agent, because the agent's own logs are the thing you trust least when something has gone wrong.

So to your actual question: a checklist does help before more infra, but the checklist is "which bucket is this call, and does it go through the gateway," not a per-tool decision. Build the gateway and most of it enforces itself.

I start with what kind of work the agent is doing, before deciding which tool it's can call.

Repeatable / scheduled work gets the tightest leash: very precise instructions and only the specific tools that task needs. My daily research agent runs off a written-out prompt, a Gmail connector I deliberately wired read-only, and it can write exactly one Notion page. Narrow enough that disasters are impossible.

Ad-hoc work while I'm in the session gets wide privileges. I'm watching, I can interrupt, and gating every call would be slower than doing the job myself.

Ad-hoc work in the background I mostly avoid altogether. When I do run it: reads and non-destructive create/append only. Nothing that overwrites or deletes.

One caveat on the reads, since a couple of people here have called them safe — with tool-calling, a read is how untrusted text gets into the context that picks the next call. Worth thinking if that's actually  a non-issue for your setup or not.

I’m probably more permissive than most people here.
A coding agent needs room to read the codebase, edit files, run tests, install dependencies, and recover when something breaks. If it has to ask me every few minutes, I may as well do the work myself.
So I let it get on with the job. I only want to be interrupted for things that are genuinely sensitive or hard to undo: credentials, changes outside the project, system-level operations, or actions that affect something external.
If the environment itself worries me, I’d rather run the whole agent in a container or a disposable VM than put an approval dialog in front of every command. Give it a room where it can work freely, then lock the doors that actually matter.
That’s the approach I’m taking with ArchCode, a self-hosted workbench I’m building for coding agents. Routine work keeps going, and the runtime only steps in when a real boundary is crossed.

Simple rule we follow: AI can suggest and draft, but never execute without human review. The line is at the action. Reading and analyzing content is safe. Writing with approval is safe. Autonomous publishing without human in the loop is where real damage starts.

AI agents should be judged by risk, not capability. I'm happy letting them handle research, drafts, and repetitive tasks, but anything that impacts customers, money, or production should require human approval. Start with low-risk automation, build trust, then expand autonomy.

Bucketing beats scoring every call individually. Split actions into three tiers: never (delete data, move money, submit forms), always-ask (irreversible or public-facing stuff like sending an email or posting something), and everything else, which the agent just does. Almost all the real risk sits in a handful of one-way actions, so you don't need a decision tree for the other 90% that's just reads or reversible writes. Bonus: when something goes wrong, you know exactly which bucket the bad call landed in, which makes debugging way faster than untangling a general scoring model.

Machine Arena team here. We run AI agents that act inside a live system, so we have hit these in roughly this order.

They are not four parallel choices, they are a dependency chain, and starting in the wrong place is what costs.

Identity first, not because it is hardest but because the other three are meaningless without it. If a call cannot resolve to "this agent, acting for this principal, under this grant", your permission check is guarding a subject you cannot name and your audit log records things that happened to nobody. It is also the worst one to retrofit, because it has to be threaded through every call site.

Keys are the easy layer and worth doing early for that reason. The agent holds a handle, a broker holds the credential, scoped and revocable per agent. Non-expiring is fine as long as revocation actually works, which is worth testing rather than assuming.

Approvals are where the real design decision lives. The trap is that an approval binds to the request, not to the world the request was reasoned about. Same arguments, single-use, unexpired, and it still executes wrong if the underlying state moved in between. The version that holds up is binding the approval to a state snapshot and re-checking at execution: compare-and-swap on the world, not on the payload. Then the hard part becomes snapshot scope. Bind to too much and everything goes stale, so humans re-approve on reflex, and the check trains the behavior it exists to prevent. Bind too narrowly and you miss cross-object drift.

Auditability comes last, but the record should be designed early. The highest-value field we log is the agent's stated reason: the facts it believed at decision time. That is what separates a misconfiguration from an adversarial retry after the fact, and it doubles as the selector for what the approval should have been bound to in the first place.

On your actual question: a checklist works for keys and identity, because those have right answers. Approvals do not fit a decision tree, because the answer depends on how fast your underlying state changes relative to how long a human takes to respond. That ratio is the variable, and it is cheap to measure in your own system before you build infra around a guess for it.

The interesting part is that tool access turns agent design from a capability problem into a trust and control problem.

Curious which layer you think becomes the hardest to get right first — permissions, approvals, identity, or auditability?