We present early results from MirrorCode, a benchmark (co-developed with METR) of long-horizon coding tasks derived from real software applications.
We find that AI models can autonomously reimplement complex existing software without access to the original program’s source code, provided there is a detailed, checkable specification.
For example, Claude Opus 4.6 successfully reimplemented gotree — a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands. We guess this same task would take a human engineer without AI assistance 2–17 weeks.
We see continued gains from inference scaling on larger projects, suggesting they may be solvable given enough tokens.
MirrorCode requires AI agents to replicate the functionalities of an existing program without access to its source code. AI solutions must match the original program’s output exactly on end-to-end tests, including held-out tests.
MirrorCode’s 25 target programs span Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.
Pkl (~61K LoC configuration language) remained unsolved at 1B token budget in early results. Claude Opus 4.7 later achieved near-perfect reimplementation of Pkl in full June 2026 release, passing 99%+ of tests.
These AI time horizons are substantially longer than METR’s estimated AI time horizon for bug-fixing and well-defined AI research engineering tasks, which is around 12 hours for Claude Opus 4.6.