How Much Autonomy Should AI Agents Have?

It’s an interesting question, let’s get into it;

Joe Kim, CEO of Druid AI, responds to the latest concerns around AI safety and the growing autonomy of AI agents.

“Predictions that AI could pose an existential threat inevitably grab headlines, especially when grounded in real incidents – like that between Open AI and Hugging Face. In fact, the inevitability of AGI is something I posted about over six months ago.

It lends to an immediate question for organisations on how we govern AI agents that are already moving beyond generating answers to taking actions.

That distinction matters as AI agents are designed to execute tasks and work across systems and processes on an organisation’s behalf. So assessing the safety of the underlying AI model is only part of the picture. Organisations also need to understand the mandate given to an agent: what it is there to achieve, what it can access, what actions it is authorised to take and where a human needs to step in.

That is why agentic AI needs clearly defined boundaries from the outset. An agent’s actions should be observable and auditable, with organisations able to measure whether it continues to operate within its intended mandate. It’s crucial we design AI Agents for behavior, not just for features and goals.

The current debate around independent testing of increasingly powerful AI models reinforces why scrutiny matters. For AI agents, testing cannot end at deployment and their performance needs to be continuously measured, with agents re-tested as models, systems and operating conditions change.

Agentic AI’s value comes precisely from its ability to act rather than simply recommend. The answer isn’t to prevent autonomy, but to make it measurable and accountable. With AI agents, trust shouldn’t simply be a claim. It should be something proven and measurable.

The AI industry spends a lot of time asking how powerful these systems could eventually become.

Organisations have two more immediate questions to answer: what power have we actually given our AI agents today and how do we know they are using it to deliver the outcomes we need?

Continuing to address this topic publicly and answer questions like these are critical to the success of this technology with humanity.”