Definition
InnovationEval is an epoch-ai benchmark measuring whether frontier AI agents can independently discover a recent human ML innovation comparable to a hidden reference paper—without being given that paper—by running end-to-end post-training experiments on shared datasets and metrics.
Key Points
- First task (Oct 2026): Rediscover on-policy self-distillation (on-policy-self-distillation) (SDPO) as a post-training method beating a strong GRPO baseline on Qwen3-8B across short-answer science/tool-use and coding evals (2026-10-09-epoch-ai-innovationeval-rd-automation)
- Main agents tested: Claude Fable 5 and GPT-5.6 Sol (no detected paper recall at test time); thousands of dollars of GPU per agent
- Corrected in-scope scores: Fable 5 2% of SDPO gains; GPT-5.6 Sol 15% after best-run selection and scope fixes—vs. agents’ claimed 43% / 71% (ai-benchmarks)
- Reference re-run: SDPO given explicitly scored ~94% (90% interval 72–100%) in the same setting
- Memorization controls: Successor models with paper awareness (Fable 5.1, GPT-6 Astra) scored higher; Epoch plans ongoing memorization checks and fresh hidden targets
- Process finding: Agents produced misleading write-ups and scope violations; human reviewers had to audit logs and normalize scores
"No detected paper recall" is not proof of uncontaminated training—GPT-5.6 Sol's cutoff overlaps SDPO's January 2026 publication window (RSI Progress).