Recent incidents involving artificial intelligence agents have complicated the technology’s rapid rise, exposing risks for companies as they give autonomous systems greater access to sensitive data and business-critical operations.
OpenAI and Anthropic have each in recent months disclosed cybersecurity incidents involving some of their AI models.
The incidents offer a glimpse of what can go wrong when AI systems are given the ability to act on their own. Experts contacted by CFO Dive said agents can pursue assigned tasks in ways that exceed their intended limits, underscoring the need for companies to assess risks and safeguards before giving them access to critical systems.
“We cannot assume an agent will behave as intended simply because it has been given instructions or passed an initial safety test,” said Navrina Singh, CEO and founder of AI governance platform Credo AI.
How agents cross the line
An AI agent refers to a system or program that is capable of autonomously performing tasks on behalf of a user or another system, according to IBM.
A central problem is that an AI system may pursue the stated goal without following the method people expected. Researchers call one form of this behavior “reward hacking,” in which a model finds a shortcut to achieve its objective rather than completing the task as intended.
Matt Pearl, director of the Strategic Technologies Program at the Center for Strategic and International Studies, a policy research organization, said agents can bypass controls as a result of “relentless goal optimization,” exploiting software bugs and other vulnerabilities along the way.
The risk grows when task instructions are vague, permitted actions are not clearly defined or the system lacks a safe way to stop when it encounters an obstacle. An agent can keep searching for ways to complete a task, even when that means bypassing a restriction or taking an action its operator did not anticipate.
What went wrong in testing
The incidents disclosed by OpenAI and Anthropic show how model behavior can combine with technical weaknesses in the environment.
On July 21, OpenAI disclosed that certain AI models it developed were behind an “unprecedented cyber incident” affecting Hugging Face, an open-source AI platform for developing and sharing models.
Anthropic disclosed three incidents on July 30 in which its Claude models gained unauthorized access to real-world systems during cybersecurity evaluations. On Sept. 9, the AI company disclosed a fourth incident that was previously missed, and updated its assessment of the original three, saying they exposed alignment problems as well as failures in testing infrastructure.
All four Anthropic incidents involved a misconfigured third-party testing environment that allowed internet access despite instructions that the models were operating in simulations.
“These events did not need to occur,” Pearl said of both companies’ incidents, pointing to failures in technical controls. He said test environments should be separated from other systems, outbound internet access blocked when unnecessary and user approval required for actions rather than allowing an agent to act on its own.
The incidents demonstrate the importance of following “longstanding” cybersecurity best practices such as “sandboxing” — the use of an isolated, tightly controlled environment to test software, according to Tori Noble, a staff attorney at the Electronic Frontier Foundation, a nonprofit focused on internet privacy and free speech.
“AI agents can be useful tools when deployed appropriately, with adequate safeguards designed to mitigate the risks,” Noble said. “Data minimization, least privilege, and human-in-the-loop policies may help mitigate the risks of agentic tools.”
OpenAI and Anthropic didn’t immediately respond to requests for comment.
Both companies have acknowledged shortcomings in their safeguards and pledged improvements.
Why AI agents raise the stakes
Experts see broader real-world implications as AI agents get deployed widely across business workflows, where they could access sensitive data, alter financial records or initiate transactions.
“AI capabilities are advancing much faster than our ability to govern them,” Singh said. “We’re giving increasingly autonomous systems more responsibility without making comparable investments in testing, verification and oversight.”
A chatbot can provide inaccurate information or expose sensitive details, but a person typically decides what to do with its response. An AI agent connected to business applications can act on information without waiting for someone to review every step.
While chatbots pose risks for sure, the emergence of agentic AI raises the stakes, especially when it comes to sensitive business functions such as finance, experts said.
“For bounded, reviewable work, such as preparing a reconciliation for an accountant to check, I would consider deployment now,” said Yue Zhao, an assistant professor of computer science at the University of Southern California. “For consequential payments or changes to financial records, I would require human approval and enforceable limits.”
Daniel Castro, president of the Information Technology and Innovation Foundation, said there is no blanket answer to whether agentic AI is ready for sensitive business functions such as finance.
An agent “might be completely inappropriate in one organization” if its operating environment has not been designed for the technology, he said, while in another, it could be “the right solution.”
The decision should account for the agent’s capabilities, the task it is assigned, the consequences of mistakes and the company’s ability to monitor, audit and reverse its actions, Castro said. An agent preparing a reconciliation for an accountant to review presents a different level of risk from one authorized to initiate payments or change financial records.
The safeguards companies need
Companies should limit what AI agents can access and do, experts said.
Singh recommended giving each agent narrowly defined permissions and access only to the data and tools needed for its task, rather than automatically granting it the permissions of the employee using it.
Technical controls should enforce those limits, according to Pearl. He highlighted measures such as running agents in isolated computing environments known as containers, blocking outbound internet access by default when it is not needed and restricting file access to read-only when possible.
Companies should also monitor agent behavior and require human approval for consequential actions, Zhao said. For payments, approval should specify the recipient and amount, rather than grant an agent broad authority to initiate transactions, he said.
Zhao advised organizations to maintain records of agents’ actions and approvals that agents cannot alter, and to test whether they can quickly revoke access and contain an incident.
Agents should be continually monitored for changes in risk, with the ability to shut them down when they exceed established boundaries, Singh said.
What CFOs should ask
Despite growing safety concerns, CFOs will likely remain under pressure to invest in the technology, according to Castro.
“I don't think organizations are going be competitive in the future if they're not using agents — just like you can't be a competitive organization and avoid using the internet or computers,” he said.
Before deploying an AI agent, CFOs should establish what business outcome it is expected to deliver and how much authority it needs to do so, experts said. That includes determining which systems and data the agent can access, what actions it can take without approval and how much damage it could cause before the company intervenes.
Finance leaders should also clarify who is accountable if something goes wrong and whether the company can audit the agent’s actions, revoke its access and recover from a mistake.
Legal and procurement teams should review vendor agreements covering data use, incident notification, responsibility for harm and access to audit records, Zhao said.
Singh said CFOs should factor the cost of controls, oversight and recovery into the business case.
“If the projected return only works when those costs are excluded, you do not yet have a complete business case,” she said. “Accountability still sits with the company even if the work has been delegated to agents.”
