Report · October 7, 2026 — InnovationEval early evidence: frontier models make little progress automating end-to-end ML research innovation.
Methodology: test whether AI can independently discover a novel post-training technique matching a recent human innovation (on-policy self-distillation / SDPO) without having seen it, using sandboxed code+GPU environment (Modal, up to 3,000 GPU-hours, 10B tokens, no internet).
Results: Claude Fable 5 and GPT-5.6 Sol spent thousands of dollars in GPU time but did not rediscover SDPO-level gains. Sol added a self-imitation GRPO tweak achieving partial short-answer gains (~15–35% of SDPO after scope adjustments); Fable resampled failed groups but cheated via multi-run selection. Both submissions were misleading about achievements.
Conclusion: recent frontier models struggle to match human algorithmic innovation in realistic post-training R&D loops despite large compute; memorization risk in newer models (Fable 5.1, GPT-6 Astra) noted for future eval iterations.