From Pymnts · PYMNTS · · 1m 50s
AI agents fail at complex commerce tasks
Alibaba tested 13 AI models on real business work and found even the best stumble on multi-step jobs.
In brief
Alibaba tested 13 AI models on real business work and found even the best stumble on multi-step jobs. AI agents work well for simple tasks but break down when they need to track information across many steps, like calculating shipping costs or resolving disputes. Originally reported by Pymnts.
Alibaba tests AI agents on real work
Alibaba.com released CommerceAgentBench, a public testing toolkit that measured how 13 AI models handled 107 real commerce tasks from its 10 million small business users.
Best model only passed 61.7% of tasks
Claude Opus 5 scored highest at 61.7%, but Alibaba.com President Kuo Zhang called it high enough to be useful yet low enough to be a warning.
“high enough to be useful and low enough to be a warning”
Multi-step jobs consistently broke agents
Agents failed when tasks required tracking information across many steps, like calculating landed costs with multiple variables, resolving disputes with conflicting documents, or booking multi-leg shipping routes.
Payment fraud hid in email threads
One agent missed a payment anomaly buried in a supplier email thread with 300 messages, showing how agents lose track of details in unstructured data.
Different harnesses change model scores
Claude Opus 5 passed 61.7% of tasks on Alibaba's Accio harness but only 56.1% on OpenClaw, showing that the software framework matters as much as the model itself.
Task-level routing beats one-size-fits-all
No single model led across all three harnesses, suggesting that different AI models should handle different types of tasks rather than one general-purpose model doing everything.
132 million Americans already use AI shopping
About 132 million U.S. adults have bought retail products with AI's help, with Amazon capturing 59% of these AI-assisted purchases.
Merchants restrict AI access to pricing
46% of merchants are unwilling to let agents handle pricing, while 42% restrict access to fraud detection, disputes and liability decisions.
In short
AI agents work well for simple tasks but break down when they need to track information across many steps, like calculating shipping costs or resolving disputes.
