AI Shopping Agents Fail Nearly Half Their Tests: The Trust Gap

Nearly half of all AI shopping agent test runs end in failure, and more than half the time the agent still completes a purchase anyway. That is not a rounding error. It is a warning shot for every Fortune 500 brand betting on agentic commerce to carry this year's holiday season.

AI shopping agents are automated systems that research, compare, and in some cases purchase products on a consumer's behalf without a human clicking "buy." Visa's newly released Visa AI benchmark just gave the industry its first hard numbers on how well these agents actually perform, and the results should alarm anyone responsible for brand experience, customer data, or holiday revenue. Visa found that when 22 artificial intelligence models from 12 labs were put through realistic shopping scenarios, 57.2% of the test runs passed and 42.8% failed .

Worse, failure does not mean the agent simply stops. In 54% of failed test runs, the AI agent still completed a purchase anyway, either choosing a product that did not meet the customer's needs or buying before gathering all requirements. That means thousands of real purchase decisions, made by a machine speaking on behalf of a customer, are landing on the wrong product, right as retailers head into their highest-stakes quarter of the year.

Matt Britton has spent his career translating emerging consumer behavior into business strategy for the world's largest brands, and he argues this moment is bigger than a technology rollout. According to Britton, agentic commerce has quietly become a trust crisis hiding inside a product launch. Brands that treat Visa's benchmark as a vendor scorecard rather than a customer experience audit are going to lose customers this holiday season, not to competitors, but to their own AI agents.

This article unpacks what Visa's benchmark actually measured, why consumers still refuse to hand over their wallets despite rapid AI adoption, and what Fortune 500 marketing and insights leaders need to do right now to make their brand data agent-legible before the holiday rush peaks. Matt Britton, a recognized AI keynote speaker, has been tracking this exact inflection point through his research at Suzy and his conversations on The Speed of Culture podcast.

What Visa's AI Shopping Agent Benchmark Actually Found

Visa did not run a marketing survey. It built a rigorous, repeatable test. The Visa Agentic Commerce Benchmark is designed to create a roadmap for industry players to deliver better agentic shopping experiences, testing AI models' ability to identify the needs of a consumer based on a conversation, uncover enough information to justify its decision to buy, and opt not to buy if nothing meets the customer's requirements.

The test scenarios were deliberately mundane and realistic. Visa put the AI models through shopping challenges that included buying a TV, a laptop, or a personal care appliance. These are not edge cases. They are exactly the category of purchases a consumer might hand off to an AI assistant this December, expecting the agent to understand specifications, budget, and personal preference well enough to buy correctly on the first try.

Performance was wildly inconsistent across providers, which matters enormously for brands choosing which agent ecosystems to optimize for. Among the best-performing AI models, 82.2% of test runs passed, while among the worst-performing models, 28.9% of test runs passed. Matt Britton notes that this gap between leaders and laggards means a one-size-fits-all approach to agent optimization is already obsolete.

Even the strongest models struggle with repeatability, which is arguably the more damaging finding for brand trust. Models passed all three attempts at a shopping scenario only 34.2% of the time. A customer who tries the same purchase through an AI agent twice could get two completely different outcomes, and neither one is guaranteed to be correct.

Cost does not fully solve the problem either, though it helps. Pass rates ranged from 28.9% for the worst-performing models to 82.2% for the best, with the five most expensive models averaging a 67.1% pass rate compared with 50.2% for the five least expensive. That seventeen-point gap is the difference between an agent brands can cautiously trust with holiday traffic and one that will generate a flood of returns.

Why 'Agentic Commerce' Is Really a Trust Crisis

Visa's own research team frames this correctly. "AI agents have the potential to power the next era of commerce, but trust is key to making it work," Rubail Birwadker, global head of growth at Visa, said in the release. That single sentence reframes the entire conversation for marketing and insights leaders. The technology is improving. The trust deficit is not closing at the same pace.

Consumer sentiment data backs this up with striking clarity. Even among consumers who had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf. Separately, a parallel holiday survey found the hesitation runs even deeper when real money is on the line.

The Talkdesk 2026 AI Holiday Shopping Survey, one of the sources behind this trend, exposes a dangerous mismatch between retailer ambition and shopper comfort. According to Talkdesk's 2026 AI Holiday Shopping Survey, 89% of retail leaders said their companies have optimised for AI agents that complete purchases on shoppers' behalf, or are actively planning or testing those capabilities, yet just 15% of US consumers would allow an AI agent to make purchases using their payment information, even with the opportunity to review its actions first. That 74-point gap between retailer readiness and consumer willingness is the real headline Fortune 500 leaders should be reacting to.

Matt Britton calls this the "agentic commerce paradox." Brands are racing to deploy autonomous purchasing capability while the very customers they are building it for are pulling back their wallets. Britton explores this exact dynamic in his book Generation AI, arguing that trust, not capability, is now the binding constraint on AI adoption across every consumer-facing industry.

Consumer fear is not primarily about the AI making a bad judgment call. It is about losing financial control. Seventy six percent of consumers named fraud or unauthorized charges as their biggest concern, more than double the 33% who worried about the AI buying the wrong item. That distinction matters enormously for how brands design agent experiences, because the fix is not just better product data. It is visible, reversible, and controllable transaction architecture.

The Cost of a Confused AI Shopper: A Brand Experience Failure

When an AI agent buys the wrong television for a customer, the brand absorbs the fallout, not the AI lab that built the model. This is the uncomfortable truth Fortune 500 marketing teams need to internalize before Black Friday. When an AI shopping agent buys the wrong television for one of your customers, who pays for the return? Visa reported that 54% of the failed runs still ended in a purchase.

Scale that failure rate across real holiday volume and the numbers get uncomfortable fast. At a 42.8% failure rate, 8,560 sessions fail, and at Visa's 54% purchase-on-failure rate about 4,620 of those end in a wrong order. Every one of those wrong orders triggers a return, a customer service interaction, and quite possibly a lost customer, all without a single human employee making an error.

Retailers are already feeling this pain in live deployments. Among the 58% of retailers that reported AI errors, 29% lost a customer as a result. That is not a technology metric. That is a churn metric, and it belongs on the desk of the chief marketing officer and chief customer officer, not buried in an IT postmortem.

Transparency compounds the problem. More than three-quarters of shoppers (77%) want to know when AI is part of their shopping experience, but 37% of retailers do not fully disclose it. Brands that let an agent transact quietly, without disclosure, and then get the order wrong are compounding a technical failure with a trust violation.

Matt Britton argues that marketing and insights leaders must stop treating agent failures as IT tickets and start treating them as customer experience incidents with executive ownership. The brands that assign clear accountability for agent-driven errors now, before peak traffic hits, will separate themselves from competitors still debating whose problem this is. This is precisely the kind of organizational readiness gap Britton addresses when he speaks to leadership teams through his keynote platform.

Making Your Brand Agent-Legible Before the Holiday Rush

"Agent-legible" is quickly becoming a required capability, not a nice-to-have. It means structuring product specifications, inventory data, pricing, and return policies so that an AI shopping agent can parse them accurately without hallucinating a feature that does not exist or missing a size restriction that does. Visa's own researchers point to the root cause of agent confusion.

"Shopping is a difficult task for AI because consumers often do not clearly know what they want at the outset, for example, they may know they want a TV, but not all the required specifications," Visa's summary of the test notes, adding that AI models are required to identify the needs of a consumer based on a conversation, and if nothing fits the customer's requirements, the AI should opt not to buy. If the AI cannot reliably extract requirements from a conversation, the brand's job is to make the product data so clean and unambiguous that the agent has no room to guess wrong.

Brands that invest in this discipline are already seeing measurable upside. Retailers using branded AI shopper agents saw 59% higher holiday sales growth in 2025, with 6.2% growth for brands with shopper agents versus 3.9% for brands without. That performance gap will likely widen in 2026 as agent traffic grows and consumer AI habits solidify.

The growth in AI-driven discovery is not hypothetical. Consumer reliance on AI assistants as the first stop in the shopping journey grew 200% in just one year from May 2025 to May 2026. Analysts expect that momentum to continue accelerating straight through the holiday quarter. One prediction holds that 20% of all 2026 holiday ecommerce traffic will originate from AI chat agents, a mix of consumer-facing bots handling live queries, autonomous agents executing backend tasks, and competitor scrapers fueling algorithmic price-matching.

Here is what Fortune 500 marketing, insights, and ecommerce leaders should prioritize before the holiday surge:

• Audit product feeds for ambiguity. Remove vague descriptors that force an agent to guess specifications instead of reading them directly.
• Publish machine-readable return and exchange policies so agents can factor friction and cost into purchase decisions, not just price.
• Decide deliberately where checkout happens. With 58% of consumers wanting the purchase to close on the retailer's own site or app, and only 12% comfortable finishing inside the AI platform, businesses have a reason to build first-party AI shopping tools on ground they already control.
• Build visible undo and review controls into any agent-assisted checkout. 84% of consumers say the ability to cancel or reverse an AI-initiated purchase after checkout is extremely or very important to them, with 64% calling it extremely important.
• Assign a single executive owner for agent-error remediation, covering returns, refunds, and service escalations, before holiday volume peaks.

Retailers are not approaching this uniformly, and that divergence is itself instructive. QVC lets agents complete purchases, Tapestry sells through Google's AI tools but keeps agents off its own checkout, and Amazon blocks outside agents while it builds its own. There is no single correct posture yet, but every brand needs a deliberate one, not a default one.

What Fortune 500 Leaders Are Getting Wrong About Agentic Commerce

The most common mistake Matt Britton sees in boardrooms is conflating AI adoption with AI readiness. Consumers are absolutely using AI throughout the shopping journey, just not in the way brands assume. Among shoppers already using AI, 73% use it as their primary tool to research products and brands.

Research is not the same as purchase authority, and that distinction is where many agentic commerce strategies are failing. A Customer Identity Trends Report found only 6% of consumers globally are comfortable allowing AI agents to make final decisions for them, like making holiday purchases, and only 4% of people said they would be comfortable letting AI make a purchase without any human review. Brands pouring budget into fully autonomous checkout flows are optimizing for a consumer behavior that barely exists yet.

Matt Britton advises clients across financial services, real estate, and retail to separate their AI investment into two distinct tracks: agent-assisted discovery, which consumers already trust, and autonomous transaction execution, which they overwhelmingly do not. Collapsing these two into a single "AI shopping" initiative is the strategic error Britton sees most often, and it is precisely the kind of nuance he unpacks for Gen Z and Gen Alpha consumer behavior on his Gen Z keynote engagements, since younger consumers are often assumed to be more comfortable with autonomous AI than the data actually shows.

The benchmark itself is still early, and Visa has been transparent about its limitations. Visa's benchmark is a lab test of 22 models on 15 shopping scenarios with three attempts each; Visa did not test checkout, payment, or order placement, and the pass rates do not describe production agent traffic. That caveat does not undercut the findings. It means the real-world failure rate, once checkout and payment friction enter the equation, could be just as volatile or worse.

Key Takeaways for Business Leaders

• Audit every product data feed now for agent-legibility, since Visa's benchmark confirms AI models still struggle to extract accurate specifications from incomplete or ambiguous listings.
• Assign executive accountability for AI agent errors before holiday volume peaks, treating wrong purchases as customer experience failures rather than isolated technical bugs.
• Build visible, reversible transaction controls into any AI-assisted checkout, since the large majority of consumers demand the ability to cancel or review AI-initiated purchases.
• Separate AI research tools from autonomous purchase tools in your roadmap, because consumer comfort with the two is dramatically different right now.
• Disclose AI involvement transparently at every touchpoint, closing the gap between what consumers want to know and what most retailers currently share.

Frequently Asked Questions About AI Shopping Agents

What percentage of AI shopping agent tests fail according to Visa's benchmark?

Visa's Agentic Commerce Benchmark tested 22 AI models from 12 labs across realistic shopping scenarios and found that 57.2% of the test runs passed and 42.8% failed . Performance varied widely by model, with the best performers passing over 82% of runs and the weakest passing under 29%.

Do AI shopping agents still complete purchases even when they fail the task?

Yes. Visa found that in 54% of failed test runs, the AI agent still completed a purchase anyway, either choosing a product that did not meet the customer's needs or buying before gathering all requirements. This means failure often results in a real, incorrect order rather than the agent simply declining to buy.

Do consumers trust AI agents to make purchases on their behalf?

Trust remains very low despite high AI usage for research. Just 15% of US consumers would allow an AI agent to make purchases using their payment information, even with the opportunity to review its actions first , and separate research puts full-autonomy comfort in the single digits globally.

What does 'agent-legible' mean for brands preparing for holiday AI shopping?

Agent-legible means structuring product specifications, pricing, inventory, and return policies so AI shopping agents can parse them accurately without guessing. Given that models pass identical scenarios only 34.2% of the time across three attempts, cleaner structured data directly reduces the odds of a wrong purchase on a customer's behalf.

The Bottom Line on Agentic Commerce Readiness

Visa's benchmark does not say AI shopping agents are failing as a category. It says the industry has rushed deployment ahead of reliability, and brands are now exposed to the consequences. Matt Britton has built his reputation helping Fortune 500 leaders see around these corners before they become headlines, and this is exactly that kind of moment.

The holiday quarter will be the largest live test of agentic commerce the industry has ever run, and the brands with clean data, clear disclosure, and visible controls will win the trust that Visa's numbers show is still scarce. Matt Britton works directly with marketing, insights, and ecommerce leadership teams to build that readiness before the stakes get higher. To bring this research and a forward-looking roadmap to your next leadership offsite or industry conference, explore Matt Britton's AI keynote speaker programs or visit his speaker hub to start the conversation today.