SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

What changed

SWE Refactor Bench is a new benchmark designed to evaluate coding agents on their ability to perform long-horizon, whole-repository software migrations. It addresses a critical limitation in existing benchmarks: the failure to verify if the migration itself has actually taken place, leading to agents passing tests by simply replicating the original code. The benchmark comprises 20 whole-repository migration tasks covering four types of technical debt. It employs a three-stage evaluation protocol: Migration Audit to confirm the migration occurred, Behavioural Tests using a fixed suite, and Agentic Verification with independent agents generating targeted tests.

Why it matters for builders

This benchmark provides a more realistic assessment of coding agent capabilities for complex software engineering tasks. It moves beyond evaluating simple code correctness to verifying the successful execution of substantial refactoring efforts. For AI builders, this means a clearer path to developing and testing agents that can tackle the significant challenge of technical debt reduction in large codebases, a common and costly problem in software development.

Practical impact

Initial evaluations using SWE Refactor Bench across 8 frontier models and 26 configurations revealed significant challenges for current coding agents. Only 5.4% of 520 runs passed all three evaluation stages, and 13 out of 20 tasks received no accepted solution. The best-performing model achieved a score of 47.0/100. The benchmark highlights a distinction between migration completeness and behavioral correctness, with many agents failing to achieve both. Performance also varies by migration category, with agents scoring lower on language rewrites compared to build toolchain rewrites.

Caveats and source limits

The provided source is a research paper introducing the benchmark. Specific details on the 20 migration tasks, the exact nature of the four kinds of technical debt, and the specific models tested beyond a general mention of "frontier models" are not fully elaborated in the excerpt. The performance metrics are based on the initial evaluation described in the paper, and further research and application of the benchmark are expected.