From arXiv · · 1m 30s
How to build better coding agents
Researchers break down the components that make AI coding agents work, revealing what actually matters for performance.
In brief
Researchers break down the components that make AI coding agents work, revealing what actually matters for performance. Context management, planning, and action space each play different roles depending on the model's strength and budget constraints. Originally reported by arXiv.
Breaking down the coding agent
Researchers tested how three key parts of coding harnesses affect AI performance: planning, action space, and context management. They ran 176 experiments across four models on real software engineering benchmarks.
Context management saves space
When memory is tight, managing context becomes critical. The best approach combines rule-based filtering first, then AI-powered summarization, preventing the agent from running out of space.
Recoverable content rarely helps
Adding the ability to recover deleted context sounds useful but doesn't improve accuracy and adds unnecessary complexity. Models simply don't use this feature.
Planning works differently by model
Weaker models use planning to improve accuracy, while stronger models use it mainly to reduce costs. Planning doesn't change how well agents perform, just where they stop working.
Tools matter for weak models
Models with poor command-line skills benefit from predefined tools. But capable models work fine with just bash access and cost significantly less, especially on terminal-heavy tasks.
Design harnesses for your model
There's no one-size-fits-all harness. The right setup depends on the model's strengths, your budget, and the task type. These findings provide a framework for matching design to needs.
In short
Context management, planning, and action space each play different roles depending on the model's strength and budget constraints.
