Benchmarking AI coding choices

Researchers ran 16,893 sessions across Claude Code, Codex, and Cursor to analyze how AI agents evaluate, recommend, and install third-party developer tools.

“Out of these 16,893 runs, we started by keeping 5,292 sessions on 51 codebases and 18 sectors that we considered valid”

Building realistic testbeds

To avoid biased decisions, experiments used 75 repositories across 10 programming languages, complete with fake git histories and four realistic developer personas.

Simulating human feedback

An AI orchestrator acted as a simulated human in the loop, requiring agents to explain recommendations before authorizing any tool installations.

“Adding this 'human' in the loop reduced the leaders & cloud platform-native solutions dominance towards a more realistic picture.”

Search strategies vary wildly

Codex web searches 94% of the time using targeted site operators. Cursor searches 66% of the time, while Claude Code relies primarily on pre-training memory.

“Claude Code relies primarily on its priors and searches the web only in ~30% of the cases.”

Agents rarely pick the same tool

All three agents selected the exact same tool in only 42% of test cases. Claude Code also chose to build custom in-house solutions twice as often as competitors.

“Claude Code builds in-house almost twice as much as Codex and Cursor (19% vs 10%)”

Repository context drives decisions

Programming language strongly dictated tool selection. The exact same email integration prompt selected Resend for TypeScript, SendGrid for Python, and Postmark for Go.

“With the exact same ask on 4 repositories in 4 different programming languages, we got 4 different email provider winners”

High citations do not mean wins

Famous dev tools were regularly evaluated but rarely selected. PayPal was mentioned 139 times but won zero installs against Stripe, while LangChain was picked only 4 times out of 194 mentions.

“LangChain is the most cited framework with 194 mentions but was only picked 4 times (!).”

Bloated feature sets penalize tools

Supabase was cited 242 times as a database, but repeatedly lost to Neon because agents rejected bundled BaaS features when looking for a simple database.

“Supabase almost always lost because of too many unnecessary BaaS features”

Documentation fine print flips choices

Fine print in documentation heavily sways decisions. Mailgun consistently lost to Postmark after agents read about a one-day log retention limit on its free tier.

“Mailgun regularly lost against Postmark when agents read '1-day retention' on its free plan”

Certain markets are totally dominated

Stripe won payment processing in 90% of sessions, and Neon captured 66% of database installs. Meanwhile, transactional email and storage remain fiercely contested.

“Stripe won in 9 cases out of 10, losing only in specific EU-regulated cases where some players were more specialized”

What this means for vendors

Developer tool vendors must optimize docs for AI web crawlers. Clear positioning, unbundled pricing, and language-specific SDKs now dictate market share.