This page may contain stale information. Last updated: 2026-06-12
Definition
NanoChat Autoresearch is Andrej Karpathy’s benchmark for automated ML research: train a small language model to the lowest validation loss (bits per byte) within a fixed five-minute budget on a single GPU.
Collaborative Extension
autoresearch@home extends the setup into a collaborative setting where dozens of humans and hundreds of agents collectively improve performance — providing a stronger comparison point than a single overnight run.
June 2026 SOTA
| System | Validation BPB |
|---|---|
| autoresearch@home community best | 0.9372 |
| recursive automated system | 0.9109 |
Recursive’s improvement equals roughly 1.3× speedup to reach Karpathy’s original overnight BPB. From a naive vanilla Transformer baseline (1.059 BPB), Recursive reached 0.9344 BPB — again beating the community (2026-06-11-recursive-automated-ai-research).
Key Discoveries
Best solutions combined architecture changes, short-context memory (hashed bigram/trigram embeddings), auxiliary losses, optimizer tuning, and compiler settings — not a single trick.