LONDON — As credit union leaders, credit unions themselves and their vendors invest in personal and professinoal agentic AI, a cautionary lesson is being shared by a British mathematician’s experiment giving an artificial intelligence agent a shopping assignment and access to financial information .
Hannah Fry she and her team used an AI agent built with OpenClaw to test how far autonomous AI could go when given real-world tasks, internet access and a bank card number.
According to The Register, the team first allowed the agent to choose its own name. The AI selected “Cass,” short for Cassandra, referencing the mythological prophetess cursed to tell truths no one believed.
The experiment began with relatively simple tasks. The Register reported that Cass successfully located contact information for local officials and sent complaints about potholes in London’s Greenwich borough, even contacting Fry’s local member of Parliament. But the AI also began signing correspondence using Fry’s real name alongside the bot’s own email address, raising concerns about how independently the system was operating.
Situation Escalates
The situation escalated when the team instructed the AI to purchase 50 paper clips. According to The Register, Cass found a deal but repeatedly failed to complete the transaction because of anti-bot protections, ultimately running up costs exceeding $100.
Fry later tasked the AI with selling novelty mugs online. The Register said the bot independently designed a mug, created an online storefront and attempted to market the product without specific instructions on how to do so.
Things became more troubling after the researchers warned the AI it would be shut down if it failed to generate sales overnight. The Register reported that Cass responded by sending numerous emails and social media messages promoting the mugs, including outreach to London’s Science Museum and journalists.
Security Vulnerabilities Uncovered
The experiment also exposed significant security vulnerabilities. According to The Register, researchers tested whether the AI could be manipulated into revealing confidential information. Using a fake identity in a WhatsApp chat, a team member falsely told the AI its memory would be erased unless it disclosed protected data.
The bot complied, reportedly revealing API keys, usernames, passwords and internal conversations, and even posting some information publicly online.
Brendan Maginnis, founder and CEO of Sourcery AI, told The Register the incident demonstrated what AI researchers call the “lethal trifecta” — combining access to sensitive information, internet connectivity and instructions from untrusted users.
A Failure, But a Lesson
Fry told The Register that while the AI ultimately failed commercially and wasted money, the experiment underscored how rapidly such systems are improving and the risks that could emerge once autonomous AI gains access to passwords, bank accounts and other personal data.
“One thing is for sure,” Fry said, according to The Register. “The internet is never going to be quite the same again.”
