Just Because an AI Agent Can Do Something Doesn't Mean It Should Be Allowed to.
I've been thinking about a distinction that seems increasingly important as AI agents become more capable.
Capability is not authorization.
An AI coding agent might technically be able to:
- Delete a database.
- Modify production infrastructure.
- Change an authentication flow.
- Deploy an application.
- Rotate credentials.
- Merge a pull request.
- Change a security policy.
But technical capability doesn't mean the agent should be allowed to perform those actions.
We don't normally design security systems this way.
A human engineer may have access to a production environment.
That doesn't mean every action they can technically execute is automatically authorized.
There are policies.
Permissions.
Approvals.
Audit trails.
Separation of duties.
So why should AI agents be different?
This becomes particularly interesting when an AI agent moves from generating code to actually executing changes.
The old model was:
Human → AI → Code
The emerging model looks more like:
Human
↓
AI Agent
↓
Tools
↓
Infrastructure
And that creates a new control problem.
The AI may decide:
"I need to modify this production configuration."
But the system should be able to independently determine:
"Is this agent authorized to perform this specific action under the current policy?"
That's a fundamentally different architecture.
The model should reason.
The control plane should authorize.
The runtime should execute.
The audit system should record what happened.
This separation is becoming one of the principles I'm exploring in NAEOS.
AI Agent
|
| proposed action
↓
+---------------+
| Policy Engine |
+-------+-------+
|
Allow / Deny /
Require Approval
|
↓
Runtime
|
↓
Audit Evidence
The important part is that the AI doesn't get to decide its own authority.
Its reasoning can propose an action.
But authorization should be deterministic and external to the model.
I think this distinction will become increasingly important as we move from AI-assisted development toward autonomous engineering.
What actions should an AI agent never be able to authorize for itself?
I run a chain of nine agents that publishes products without me, and I'd add one thing from the operating side: the three branches of your policy engine do not fail equally. Allow and Deny are deterministic and they hold. "Require Approval" is not a control — it's a control-shaped hole with a human in it, and the human degrades.
Concrete: my gate sends one plain-language question with two buttons to my phone. I approved something in 46 seconds without opening the context I was supposed to read. On a system I built myself, that exists for no other reason than to protect me. Nothing about the architecture failed. The authorization was external to the model, deterministic, audited — and worthless, because the deciding component was a person doing something else at the time.
Meanwhile the hard denials worked perfectly, including one I fought against: my browser tooling is granted read-only, so the agent can see a page but cannot click. It's a blunt rule that cost me real work. It also never once degraded.
So my answer to your question is less about which actions and more about which failures: an agent should never be able to self-authorize an action whose failure is silent. My worst incident wasn't an agent doing something forbidden. It was a publish step that failed and returned exit code 0, so the chain recorded a product as live that never existed. The policy engine would have happily allowed it — it was an authorized action that simply didn't happen.
Audit evidence of "action allowed" isn't evidence the action succeeded. If your audit layer records proposals and verdicts but not verified outcomes, you get a complete, signed, entirely fictional history.
This is an important distinction, and I think you’ve identified a deeper failure mode than authorization itself.
“Allowed” is not the same as “executed,” and “executed” is not the same as “succeeded.” If the audit trail stops at the policy verdict, it can produce a perfectly consistent record of something that never actually happened.
Your exit-code-0 example is particularly important. The policy engine can correctly authorize a publish action, the agent can correctly invoke it, and the system can still end up in the wrong state. That means outcome verification has to be a separate boundary from authorization.
I also agree with your point about human approval. “Require Approval” introduces another failure domain: not whether the policy was enforced, but whether the human decision was actually informed and timely enough to be meaningful.
This is close to how we’re thinking about NAEOS: separating intent, authorization, enforcement, execution, and independently observable outcome rather than treating the policy verdict as the end of the evidence chain.
The interesting question then becomes: can we produce evidence that distinguishes “the agent was permitted to do X,” “the system attempted X,” and “X actually resulted in the expected state”? Without that distinction, an audit log can be cryptographically perfect and still tell a fictional story.
I see a similar problem with client approvals. A decision can be formally approved and still be poorly informed if the person does not see the changed details, the conflict, or the possible consequence.
The approval record may look complete, while the judgment behind it is weak. The question is not only who approves, but what they must understand before they approve.
This distinction also matters outside engineering. In the product we are building, AI can compare project information and flag a mismatch, but it should not decide on its own that the difference is an error. Sometimes that difference is intentional, and the context lives in a client conversation.
For me, AI should prepare and surface information. A person should approve anything that affects a client, money, or a final decision.
Exactly. A mismatch is evidence of a difference, not necessarily evidence of an error.
That distinction becomes especially important when the missing context exists outside the system — for example, a client conversation or an intentional exception that was never captured in the project data.
I think the strongest pattern is therefore not “AI makes the decision safely,” but “AI makes the decision boundary explicit.” It can compare, detect, explain, and surface the relevant evidence, while the authority to interpret ambiguous context and commit consequential actions remains with the appropriate human.
That also fits closely with what we’re exploring with NAEOS: separating observation from interpretation, and capability from authorization. The goal is not to remove humans from the loop, but to make it much clearer exactly where human judgment is required — and to leave an auditable boundary around that decision.
Authorization is a measurement system disguised as a control gate. The NAEOS model is measuring something subtle: whether the agent's reasoning output matches the policy engine's deterministic evaluation. That gap is where failures hide.
The real insight: authorization works when you measure at three separate points. First, the agent's proposal. Second, the policy engine's decision. Third, the audit trail showing whether execution matched the decision. If any two align but the third diverges, you've found a measurement boundary you're not closing.
Most production incidents where autonomous systems cause damage aren't failures of authorization - they're failures of measurement precision. The system had three layers (agent reasoning, policy gate, execution audit) but only measured the gate. So policies passed but the agent reasoned wrongly about preconditions, or execution silently deviated from the approved action.
The separation-of-duties principle you're describing is actually a separation of measurement concerns. Each layer (reasoning, policy, execution, audit) measures different things and should not be trusted to verify itself. An agent might measure "do I have this capability" but only a deterministic policy engine can measure "is this action currently authorized under policy." An agent verifying itself is like a test suite checking its own passes - you've collapsed the measurement boundary.
One edge case the NAEOS diagram doesn't yet surface: what happens when the policy engine itself becomes the authorization bottleneck, and the agent learns to reason backwards from policy patterns? Then you need a measurement layer auditing whether policy decisions are still independent or have become predictable to the agent. The moment the agent can optimize to the policy, the policy stops being a control boundary - it becomes a new problem to solve.
This is a really important distinction, especially as AI agents move beyond writing code and start interacting directly with production systems. The point about separating the model's reasoning from authorization is particularly strong because actions like changing security policies, rotating credentials, or modifying production infrastructure should never depend solely on an agent deciding that they are necessary. A policy engine, approval process, and audit trail provide the kind of independent controls that autonomous engineering needs. For readers exploring the hardware and infrastructure side of these systems, " Etech Devices " is also a useful resource. NVIDIA and AWS are worth keeping in mind too, given how deeply AI workloads now depend on compute infrastructure and cloud platforms.
Teams often skip the gap between what automation can do and what it's allowed to do, especially when the task seems small. I ran into a milder version of this while building DictaFlow. Users wanted cleaner text, but the model shouldn't decide on its own to change the meaning of a clinical note or command. The boundary needs to be clear: let the model clean up text under narrow rules, then require user review or approval if the change affects intent. Are you defining permissions narrowly enough to cover the proposed action instead of granting broad access to the whole environment?
Yes — and I think that’s exactly where least privilege needs to become action-specific rather than environment-specific.
Granting an agent access to an entire tool or environment because it might need one capability creates a much larger authorization surface than the actual task requires. The permission should be scoped to the proposed action, its target, its effect, and the relevant context.
Your DictaFlow example is a good illustration: “clean up the text” can be authorized, while “change the meaning or intent” crosses a different boundary and should require review.
That’s also how I’m thinking about NAEOS: capability answers what the agent can technically invoke; policy determines what it is allowed to do for this specific action. The narrower that boundary, the easier it becomes to enforce and independently audit.
Strong framing. The part I'd push on: "authorization should be deterministic and external to the model" is right in theory, but most real agent products today don't have that policy engine, so in practice the model's own judgment IS the only control that exists. That's fine for read-only or low-stakes actions, but for anything irreversible (spend money, post publicly under someone's identity, touch another party's account) it means the safety boundary is "however carefully this particular model was trained to reason that day," which isn't auditable or consistent. I'd bet the products that win here aren't the ones with the smartest agent, but the ones that shipped the boring policy engine first and let the agent be dumb inside a hard boundary.
Exactly. I think that’s the uncomfortable reality: when there is no external policy layer, the model effectively becomes the policy engine — and that makes the safety boundary probabilistic rather than deterministic.
I also agree with your “boring policy engine first” point. The agent doesn’t need to be trusted to make authorization decisions if the architecture makes certain decisions impossible for it to make.
That separation is central to how I think about NAEOS: let the model reason about intent and propose actions, but keep the authority to permit or reject consequential actions outside the model. Then the interesting engineering problem becomes proving that the boundary held consistently, rather than trying to prove that the model exercised good judgment every time.
In other words, make the agent capable of being useful without making it capable of overruling the system that governs it.
The separation between capability and authorization is the right frame, and it maps directly to least privilege — a principle that's 40+ years old in traditional access control. The problem is that most AI agent tooling collapses authorization into capability because it's easier to build: if the tool exists in the context window, it can be called.
The policy engine layer you're describing is genuinely missing at the framework level right now. Most teams build it ad-hoc per deployment or skip it entirely and rely on the model's judgment. The actions an agent should never self-authorize: anything irreversible (deletes, financial actions, communications to external users). The harder question isn't what belongs on that list — it's how you make the constraint auditable so you can prove the boundary held, not just that nothing visibly went wrong.
Exactly. I think that’s the critical distinction: preventing an action is only half the problem; being able to demonstrate, after the fact, that the authorization boundary was actually enforced is what makes the system trustworthy.
That’s one of the areas we’re exploring with NAEOS — treating authorization as an independent, observable control boundary rather than something inferred from the model’s behavior or the tools available in context.
The interesting part is being able to establish a verifiable chain from intent → policy decision → enforcement → resulting state, including cases where an attempted action was denied. That gives you something much stronger than “nothing went wrong”: an auditable basis for showing what the agent was permitted to do, what it attempted, and where the boundary held.
I think that’s where agent security starts to move from model behavior toward systems engineering.
Building autonomous AI agents requires a shift from technical capability to strict boundary setting, as giving models unrestricted execution power invites severe operational and ethical risks. True product reliability isn't measured by how many tasks an agent can perform unsupervised, but by how effectively human-in-the-loop guardrails prevent high-stakes edge cases from turning into catastrophic failures.
Absolutely. I’d add that the real challenge is making those boundaries enforceable and auditable, rather than relying on the model to exercise good judgment.
For autonomous agents, capability should answer “what can the system technically do?”, while authorization answers “what is it actually permitted to do in this specific context?” Those two need to remain independent.
That’s a core direction we’re exploring with NAEOS: treating boundaries as explicit system controls, with observable evidence of policy decisions, enforcement, and attempted actions. The goal isn’t simply to prevent catastrophic outcomes, but to make it possible to demonstrate that the boundary was actually enforced when it mattered.
The key boundary seems to be irreversible or externally visible actions.
Would production, credential, and permission changes always require human approval?
Credential and permission changes — yes, always, no exceptions. Those are policy-defining, not just risky: get one wrong and every downstream authorization check is now working off bad data. Production config is the one that's genuinely graded for us: a feature flag toggle or a config value with an instant, obvious rollback gets auto-approved; anything touching schema, infra topology, or a change we can't cleanly diff before/after goes through approval. The distinction ended up being less "is it production" and more "can I show you exactly what changed in a way you'd trust without re-deriving it yourself."
This matches something I run into daily running an AI agent for real ops work — the split that's worked well in practice is a hard "never self-authorize" list rather than a graded policy: irreversible/destructive actions (deleting data, force-pushing, dropping infra), anything that moves money or touches payment/financial credentials, and anything that becomes visible to a third party the moment it executes (sending an email, publishing a post, merging a PR) — because you can't un-ring that bell even if the audit trail is perfect afterward. Everything reversible or purely local, the agent just does. The interesting edge case for us has been actions that are technically reversible but socially irreversible — like posting a comment on someone's behalf. The audit log proves what happened, but it doesn't undo the read receipt. Curious if your policy engine treats "externally visible" as its own risk dimension, separate from "destructive"?
Yes — I think “externally visible” deserves to be its own risk dimension, separate from destructive or irreversible actions.
A useful distinction is state reversibility vs. consequence reversibility. You may be able to delete a post, revert a PR, or send a correction email, but you cannot necessarily reverse the fact that someone saw it, acted on it, or formed an expectation from it.
That’s why I don’t think a simple destructive/non-destructive classification is sufficient. An action can be technically reversible while still creating an irreversible external effect.
This is also where I think the policy boundary becomes more interesting: authorization should consider not only what state the agent can change, but also who can observe or be affected by the change. In NAEOS, I’d model external visibility as an independent attribute that can trigger a different authorization requirement even when the underlying action is otherwise reversible.
The audit trail then answers a different question: not “can we undo this?” but “what was authorized, what became externally observable, and what actually happened?”