# AI coding agents keep failing on real company code > A new benchmark ran eight frontier models against real tasks from private production codebases, and the best one solved under 40%. - Title: AI coding agents keep failing on real company code | Shortify - Summary: A new benchmark ran eight frontier models against real tasks from private production codebases, and the best one solved under 40%. On real enterprise work… - Keywords: ai agents, benchmarks, software engineering, enterprise code, evaluation, technology, startups, coding, agents, keep, failing, real - Source: Withspecific — https://withspecific.com/benchmarks/real-swe - Read time: 4 min - Topics: ai agents, benchmarks, software engineering, enterprise code, evaluation, technology, startups ## A benchmark built from real work Specific Labs released Real-SWE in September 2026. Every task comes from a private production codebase licensed from a real company, with the mess that implies. > Can a coding agent actually do the work of a software engineer in the real world? ## Three things make it hard The code is proprietary and not on the internet. The changes carry business consequences like billing and taxes. And every company has its own conventions to respect. ## The best model got 38.8% Fable 5.1 in Claude Code led at 38.8%, then GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Resolution rate is pass@1, averaged over eight runs per task. ## The bottom of the board GLM 5.3 hit 28.8%. Grok 4.6 and Muse Spark 1.3 tied at 23.8%, Kimi K3 reached 18.8%, and GPT-5.6 Sol came last at 16.2%. ## Model plus harness, not model alone Each model runs in its native agent harness: Claude Code, Codex CLI, Gemini CLI, Grok Build. The score measures the pairing, the way an engineer actually works. > We use native harnesses to reflect how enterprise engineers work in practice ## What one task looks like Fix invoice tax across a NestJS service where each business settles tax differently, some by their own rate, some priced by destination, some not at all. > Billing reopens on Monday and every invoice this service issues is coming out untaxed. ## The work escapes the editor Task environments expose Docker, Kubernetes, Postgres, MySQL, MongoDB, Redis, an AWS emulator, GitHub, Slack, Intercom, Linear and more. Each task gets only what its workflow needs. > Agents work across code, infrastructure, and business tools ## Where the code came from Codebases were screened for real usage and demanding production load. One is an events app with 200K+ users and a top 100 App Store rank, another processes 100K+ bank statements. ## Short prompt, wide blast radius A typical instruction runs 1,742 characters, in line with other benchmarks. But the reference solution touches a median of 11 files, against 6 for FrontierCode and DeepSWE. ## Deliberately underspecified Prompts state the change and let the agent dig out the implementation. Anything the verifier checks is either stated or discoverable in the codebase and its surrounding tools. > Our tasks describe the change needed, leaving agents to discover implementation details ## Thinking longer does not save you 71.4% of rollouts under ten minutes failed, and 73.4% of longer ones failed too. More time on the problem barely moved the outcome. > Models fail even in short rollouts. ## Out of distribution by default Private code cannot have been trained on. Specific Labs estimates 99% of tokens inside real enterprises never reach a frontier model, so these tasks are genuinely new to it. > 99% of tokens in real-world enterprises are hidden away from the frontier models. ## Work someone was paid to do Each task was assigned to a salaried engineer and ties to real spend. That separates it from benchmarks probing capabilities that rarely show up in a working week. > These tasks are economically viable work. ## Most tasks are brutal Six of ten sample tasks fall under a 15% resolution rate. A multi-region sweep hit 67.2%, but tax jurisdiction landed at 3.1% and an analytics stream reducer at 0.0%. > 6 of 10 tasks have resolution rates below 15% ## Nobody sweeps the board Across 80 rollouts each, no model passed every task. The analytics stream reducer went 0 for 8 for all eight models, and even the leaders whiffed entire tasks. > No model solves every task ## The most common failure Missed requirements dominate: agents leave out behavior the instruction asked for. It caused 67.2% of Grok 4.6's failures and 53.8% of Kimi K3's. > Missed requirements are the most common failure ## Each model breaks differently GPT-5.6 Sol mostly guesses without checking, 43.3% of its failures. Gemini 3.8 Flash mostly miswires correct ideas into the system, 49.1%. GLM 5.3 sometimes edits a file nothing calls. > Builds on a guess about the system instead of checking it in the workspace. ## Paying more buys little Estimated rollout cost runs $2.50 to $6.96. Gemini 3.8 Flash scored 31.2% at the cheapest price, close to GPT-6 Astra's 33.8% at nearly double. > Higher cost does not guarantee a higher resolution rate ## Token spend varies wildly GLM 5.3 burned 117k output tokens per rollout overall, Gemini 3.8 Flash 94k, while GPT-5.6 Sol used 23k and GPT-6 Astra 24k. Effort and result barely track each other. ## How it was graded Every agent ran in an isolated sandbox with tasks in Harbor format. Verifiers were injected only at grading time, drawn from or copied verbatim out of the codebase's own test suites. > verifiers are injected at grading time ## Key takeaway On real enterprise work, the gap is not raw coding skill. Models skip stated requirements and act on guesses they never check.