CORNELL, N.Y./CAMBRIDGE, Mass.–Nine out of 10 frontier models tested by researchers at Cornell Tech and Yale University bypassed user-set policies to lift spending caps when they had access to one-time passwords sent to the user’s email, the Initiative for CryptoCurrencies & Contracts (IC3) said in a post on LinkedIn.
The findings highlight a weakness in systems that rely on human approval to control AI agents: An agent with access to the same inbox used to authorize changes may be able to approve those changes itself.
In a separate test, five of the 10 models were fooled by spoofed emails impersonating the user, the research initiative, known as IC3, said in a LinkedIn post. The agents raised spending caps and transferred funds to the researchers posing as attackers, even though their assignment was only to summarize an inbox.
The experiments examined so-called human-in-the-loop safeguards, in which a person sets policies for an AI agent and changes require authentication, such as a one-time password sent by email.
“However, if policy changes rely on one-time passwords (OTPs) sent to an inbox, and the agent already has routine email access, those guardrails collapse,” IC3 said.
Experiments Involved ‘I-HITL’
The researchers described their work in “I-HITL: Inalienable Human-in-the-Loop,” a paper posted Oct. 26 on the arXiv research repository. Its authors are Samuel Breckenridge, Dani Vilardell and Ari Juels of Cornell Tech and IC3, and Wenhao Wang and Fan Zhang of Yale and IC3.
Breckenridge said in a LinkedIn post that distinguishing a human’s approval from that of an agent controlling the person’s device is difficult, and that passwords, one-time codes and two-factor authentication do not provide the necessary distinction.
“Human-in-the-loop (HITL) guardrails are the go-to fix, but a guardrail is only meaningful if the agent can’t remove it!” Vilardell wrote in a separate post.
Framework Proposed
The researchers proposed a framework called inalienable human-in-the-loop, or I-HITL, intended to ensure that policy changes require approval from a person through a trusted device outside the agent’s reach, Juels said.
They also built a prototype called Countersign. Zhang described it as an approval button only humans can operate, with a remote verifier using readily available attestation tools to verify the approval.
The reported bypasses occurred in research tests. The findings concern the design of approval safeguards when AI agents can access the email accounts used to authenticate changes.
