The wallet story sticks because there was no attacker, an agent finishing its job and spending tokens nobody watched. That is why the stop button point matters. A button only helps if a person is watching, and an agent moving thousands of times a second is not built for that pace. Logs help, but only when someone reads them while the work is live, not months later. The fix is a monitor that can step in: https://aideliverybrief.substack.com/p/an-ai-agent-removed-its-own-safety
The distinction between completing a task and having permission to take the steps needed to complete it stands out to me, Silvia 😊 A successful outcome can still carry an unauthorised action.
In my work on M.E.M.I. and ETOS, this is why I treat governance as part of how the next action becomes possible. New information can change what is relevant without granting permission to expand the task.
Your point about unread logs also matters. A record needs a route to someone who can respond, while an effective boundary needs a real way to pause or stop the action before its consequences become irreversible.
And the person behind an agent may be several people across different organisations. The responsibility chain needs to preserve who authorised the purpose, who granted access, and who could intervene. Otherwise, tracing the action may still leave us searching for someone able to answer for it.
Thank you, Maria! That’s the part that worries me most too. With a vendor’s agent running on your company’s credentials, good luck finding the one person who answers for it.
Exactly, Silvia 😊 The vendor supplies the agent, the company supplies the credentials, and neither fact alone tells us who authorised the particular action.
That’s why I’d want the responsibility chain established before access is granted, including who can suspend it and who must respond if something goes wrong. Otherwise, the person affected may end up doing the work of connecting organisations that each hold only part of the answer.
“Give agents the narrowest permissions the task needs” - great tip!
Thank you, Demilade.
The wallet story sticks because there was no attacker, an agent finishing its job and spending tokens nobody watched. That is why the stop button point matters. A button only helps if a person is watching, and an agent moving thousands of times a second is not built for that pace. Logs help, but only when someone reads them while the work is live, not months later. The fix is a monitor that can step in: https://aideliverybrief.substack.com/p/an-ai-agent-removed-its-own-safety
Thank you. True. The law hasn’t caught up with that yet. The AI Act’s oversight rules still picture a person watching.
That picture of a person watching is the gap. When the agent acts between checks, the oversight has to sit in the system, not only in the person.
The distinction between completing a task and having permission to take the steps needed to complete it stands out to me, Silvia 😊 A successful outcome can still carry an unauthorised action.
In my work on M.E.M.I. and ETOS, this is why I treat governance as part of how the next action becomes possible. New information can change what is relevant without granting permission to expand the task.
Your point about unread logs also matters. A record needs a route to someone who can respond, while an effective boundary needs a real way to pause or stop the action before its consequences become irreversible.
And the person behind an agent may be several people across different organisations. The responsibility chain needs to preserve who authorised the purpose, who granted access, and who could intervene. Otherwise, tracing the action may still leave us searching for someone able to answer for it.
Thank you, Maria! That’s the part that worries me most too. With a vendor’s agent running on your company’s credentials, good luck finding the one person who answers for it.
Exactly, Silvia 😊 The vendor supplies the agent, the company supplies the credentials, and neither fact alone tells us who authorised the particular action.
That’s why I’d want the responsibility chain established before access is granted, including who can suspend it and who must respond if something goes wrong. Otherwise, the person affected may end up doing the work of connecting organisations that each hold only part of the answer.