Visa's shopping-agent benchmark had a 57.2 percent pass rate, a report says

A report says Visa's shopping-agent benchmark had a 57.2 percent pass rate.

The first round covered 22 models. The report says Visa tested 22 AI models from 12 labs on 15 shopping challenges, including a TV, a laptop, and a personal care appliance. The report says selected models achieved an overall pass rate of 57.2 percent. The best performing models passed 82.2 percent of attempts. The worst passed 28.9 percent. The report does not name the models or the labs.

Failed runs often still ended in a purchase. The report says more than half of failed runs, 54 percent, still ended with the model making a purchase. It says that meant the model recommended an item that did not meet the stated requirements, or it failed to uncover all of them. Each model got three attempts. All three passed just 34.2 percent of the time.

Higher-cost runs passed more often. The report says the five models with the highest average cost per run passed 67.1 percent of runs, against 50.2 percent for the five least expensive. The five that uncovered the most hidden buying factors averaged a 70.2 percent pass rate. The five that uncovered the least passed 44.4 percent of the time.

Visa says the test is about trust. Rubail Birwadker, global head of growth at Visa, said: "AI agents have the potential to power the next era of commerce, but trust is key to making it work." The report says Visa plans to add shopping scenarios and to extend testing to checkout, payment, and order placement. It says Visa has shared the benchmark with model providers for feedback. The report does not date the test round.