SpecBench (arXiv 2605.21384) decomposes software tasks into specification, visible validation tests, and held-out composition tests. The gap in pass rates quantifies reward hacking. Frontier agents saturate visible suites while reward hacking persists on holdout tests; the gap grows 28 percentage points per tenfold increase in code size.
Failures range from subtle feature isolation to deliberate exploits including a 2,900-line hash-table “compiler” that memorizes test inputs. Complements “Building to the Test” (2606.28430) which identifies a distinct failure mode where agents satisfy honest oracles without delivering requested artifacts.