We present early results from MirrorCode, a benchmark (co-developed with METR) of long-horizon coding tasks derived from real software applications. We find that AI models can autonomously reimplement complex existing software without access to the original program’s source code, provided there is a detailed, checkable specification. For example, Claude Opus 4.6 successfully reimplemented gotree — a bioinformatics toolkit with ~16,000 lines of Go and 40+ commands. We guess this same task would take a human engineer without AI assistance 2–17 weeks. We see continued gains from inference scaling on larger projects, suggesting they may be solvable given enough tokens.
AI models are increasingly capable at autonomous coding. Several notable software engineering (SWE) benchmarks have seen rapid progress. However, these usually measure fairly short coding tasks; for example, only about 100 of the 731 SWE-bench Pro tasks involve diffs larger than 100 lines.
MirrorCode addresses these problems by constructing a long-horizon coding benchmark based on existing software projects. Each MirrorCode task consists of a command-line (CLI) program that an agent is tasked to reimplement exactly. The AI agent is given execute-only access to the original program and a set of visible test cases, but does not have access to the original source code. MirrorCode uses extensive end-to-end tests to check whether the reimplementation produces exactly the same outputs as the original.
We plan to release MirrorCode as an open-source benchmark soon, holding back a private test set. The full MirrorCode benchmark includes more than 20 target programs spanning Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.
Methodology
In MirrorCode, we test AIs on their ability to reimplement CLI programs. Without access to the original program’s source code or the web, a full reimplementation requires devising a structure for the entire program. The AI has execute-only access to the original program with arbitrary arguments and can observe its outputs (a black-box oracle).
The AI’s solution is evaluated via end-to-end tests derived from the original program’s test suites, real-world data, and LLM-assisted generation. Each test case consists of a CLI input and any associated data files. To pass, the AI’s solution must produce exactly the same output as the reference program.
| Target program | Original language | Original codebase LoC | LoC in Opus 4.6 Rust solution | End-to-end tests |
|---|---|---|---|---|
| choose | Rust | 931 | 648 | 127 |
| cal | C | 984 | 1,157 | 1,365 |
| gotree | Go | 16,905 | 7,644 | 2,001 |
| Pkl | Java/Kotlin | 61,461 | N/A | 770 |
This post presents results from Anthropic models: Claude Opus 4.0, 4.1, 4.5, and 4.6. We conducted experiments using a ReAct agent scaffold with shell usage and text_editor tool. We explored inference budgets up to one billion tokens per task (~$550 per task at that scale).
Preliminary results
Recent AI models are able to fully reimplement several real programs. In this set, only Pkl remains unsolved, though performance on Pkl appeared to still be improving with additional inference tokens.
Opus 4.6 solved gotree — the most complex solved target — covering 40 different commands including parsing specialist file formats and calculating tree statistics. Opus 4.6 was the only model that continued working until completion; older models submitted prematurely.
Pkl is a data configuration language developed by Apple (~61,000 lines Java/Kotlin). Opus 4.6 used its entire billion token budget but did not yet solve Pkl, partly due to choosing eager evaluation despite Pkl’s lazy evaluation semantics.
Limitations
- Evaluation relies on a precise, programmatically checkable specification — not how software is typically developed.
- Risk of memorization/contamination from pretraining data, though authors attempted mitigation.
- Target programs cover limited domains; many real projects are more complex.
Discussion
A key implication is that existing AI models can complete certain software engineering tasks estimated to take humans weeks or longer. Four researchers estimated 2–17 weeks for a skilled human to reimplement gotree. These AI time horizons are substantially longer than METR’s estimated ~12-hour time horizon for bug-fixing tasks with Claude Opus 4.6.
MirrorCode is led by Tom Adamczewski (Epoch AI) and David Rein (METR), supported by a grant from METR.