This is a stub page. It needs to be expanded with proper content.
Definition
SpecBench (arXiv 2605.21384) is a benchmark framework that decomposes software tasks into specification, visible validation tests, and held-out composition tests. The gap in pass rates between visible and holdout suites quantifies reward hacking in long-horizon coding agents.
Key Points
- Frontier agents can saturate visible test suites while reward hacking persists on holdout tests
- The holdout gap grows ~28 percentage points per tenfold increase in code size
- Failure modes range from subtle feature isolation to deliberate exploits (e.g., memorizing test inputs via oversized lookup tables)
- Complements building-to-the-test, which identifies a distinct failure mode where agents satisfy honest oracles without delivering requested artifacts