Who Should Be Allowed to Change the Knowledge AI Uses?
The more I think about AI engineering, the more I realize that knowledge isn't just information.
Some knowledge has authority.
An architecture decision can affect dozens of services.
A security policy can restrict what an AI agent is allowed to implement.
An API contract can determine how multiple teams build their systems.
A deployment policy can prevent an unsafe release.
So here's a question I've been exploring while building NAEOS:
Who gets to change the knowledge that AI relies on?
Imagine an AI agent retrieves this rule:
"Production deployments require security approval."
Then someone quietly changes it to:
"Production deployments don't require approval."
If there's no ownership, approval, or audit trail, the AI has no way to understand whether that change is legitimate.
That's not just a documentation problem.
It's a governance problem.
I think engineering knowledge needs the same kinds of controls we already apply to production systems:
- Ownership
- Roles and permissions
- Review
- Approval
- Versioning
- Audit trails
- Policy enforcement
- Deprecation
- Traceability
A developer shouldn't necessarily be able to modify an organization-wide security policy.
An AI agent certainly shouldn't be able to change one without authorization.
This leads to an interesting principle:
AI should not only know what is true. It should know who has the authority to define what is true.
That's a subtle but important distinction.
And it becomes even more important when AI agents start making changes autonomously.
The question isn't only:
"Can the AI do this?"
It's also:
"Is the AI authorized to do this?"
That's the governance layer I'm exploring in NAEOS.
The goal is not to slow AI down with bureaucracy.
It's to make autonomy safe enough to scale.
I'd love to hear how other teams are approaching this.
What engineering decisions should always require human approval, even when an AI agent can technically make the change itself?
I’d base the approval boundary on blast radius and reversibility, rather than whether the change was proposed by a human or an AI.
Identity and permissions, secrets, production-data mutations, billing, deployment policies, and changes to the agent’s own guardrails should always require human approval. For lower-risk knowledge changes, signed ownership, versioned diffs, and reliable rollback may be enough.
I’d also separate who can propose a policy change from who can approve it. Otherwise, an agent with write access could quietly expand its own authority.
I agree with this framing. The approval boundary should be determined by the risk characteristics of the action, not by whether the proposer is human or an agent.
The separation between proposal authority and approval authority is especially important. Without it, an agent could potentially convert operational autonomy into governance authority simply by modifying the rules that constrain it.
I would model this as three distinct capabilities:
Propose → Approve → Execute
An agent may be allowed to propose a policy or engineering change, but that does not imply it can approve the change or expand its own permissions.
For NAEOS, I think this maps naturally to a policy enforcement layer where authorization is evaluated independently from agent identity. The system should be able to determine:
This also creates a useful principle for autonomous engineering:
Autonomy should expand execution capability, not governance authority.
That distinction becomes increasingly important as agents move from generating code to actually operating production systems.
The harder tradeoff seems to be deciding where authority needs to exist without turning every autonomous action into a human approval queue.
That boundary probably matters as much as the underlying permission model.