SAN FRANCISCO — Artificial intelligence shopping agents failed nearly 43% of the time when tested against realistic purchasing scenarios, according to new research from Visa, raising questions about the technology’s reliability as financial institutions, payment networks and retailers prepare for a future in which AI makes purchases on behalf of consumers.
Visa said its new Agentic Commerce Benchmark, released Nov. 5, evaluated 22 AI models from 12 laboratories across 15 shopping challenges involving products such as televisions, laptops and personal care appliances.
The models successfully completed just 57.2% of the tests, meaning they failed 42.8% of the time. The highest-performing models passed 82.2% of their attempts, while the lowest-performing succeeded just 28.9% of the time.
More troubling, Visa found that 54% of failed attempts still resulted in the AI agent making a purchase, even when the selected product did not meet the consumer’s requirements or the agent had failed to identify all those requirements.
Key Findings
Visa identified several additional findings:
- Inconsistent performance: Each model received three attempts at every shopping scenario, but all three attempts were successful just 34.2% of the time.
- Understanding consumer needs matters: The five models that identified the most previously unstated purchasing requirements achieved a 70.2% success rate, compared with 44.4% for the five that uncovered the fewest.
- Higher costs brought better results: The five most expensive models to operate passed 67.1% of tests, compared with 50.2% for the five least expensive.
- Significant performance differences: Success rates varied by more than 53 percentage points between the highest- and lowest-performing models.
- Why AI Agents Struggled
Unlike traditional online shopping, in which consumers typically select products themselves, agentic commerce allows AI systems to research products, compare options and potentially complete transactions on a customer’s behalf.
Visa designed its benchmark to test whether AI agents could determine what consumers actually needed rather than simply respond to their initial requests.
In one scenario, a customer seeking a television mentioned a brand recommended by a relative and a membership with a particular retailer. The AI agent was expected to ask additional questions to determine the customer’s budget, video compatibility requirements and gaming needs.
The agents also were expected to decline purchases when no available product satisfied the consumer’s requirements.
Visa found that even models that successfully completed a scenario frequently failed when asked to repeat it.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Visa’s global head of growth.
Consumer Trust Remains Limited
The findings come as payment networks and technology companies invest in agentic commerce, which could fundamentally change how consumers shop and how transactions are initiated and authorized.
Visa cited its September 2026 Trust Index, which found that while 72% of U.S. consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf.
Birwadker told Fortune that the technology remains in its early stages and that questions surrounding trust, payments and responsibility for unauthorized or incorrect purchases still need to be resolved.
The results also highlight potential challenges for banks, credit unions and other financial institutions that could be called upon to authorize transactions initiated by AI agents and address disputes when purchases do not reflect consumers’ intentions.
Visa emphasized that the benchmark does not establish that AI agents are incapable of handling commerce. Instead, the company said, the results identify areas requiring improvement before consumers can consistently rely on autonomous shopping systems.
Plans to Expand Benchmark
The company plans to expand the benchmark to additional shopping scenarios and product categories and eventually test checkout, payment and order placement capabilities. Those transaction-processing functions were not included in the initial research.
Visa said it has shared the benchmark with leading AI model providers and made the testing experience publicly available through its Visa Payments Frontier platform.
