First Steps Toward Automated AI Research
Early results from Recursive’s automated AI research system on model training and GPU kernel benchmarks
JUNE 11, 2026
Today we are releasing early results from Recursive’s automated AI research system. Across three benchmarks, the system achieves state-of-the-art results: in fixed-budget language model training, small-model training speed, and GPU kernel optimization.
The system automates the research loop for a target objective: it proposes an idea, implements it, runs an experiment, validates the result, and uses what it learns to choose the next experiment. It runs many research threads over long horizons, keeps useful context from prior experiments, combines promising branches, and puts results through validation for reward hacks and variance before treating improved performance as real progress. It is designed to scale and harnesses principles of open-ended algorithms, building on ideas from previous work by our team and others into recursively self-improving AI.
We tested the system on benchmarks chosen for both practical importance and tight feedback loops. They stress three core levers of AI progress: better training algorithms, faster training, and more efficient use of hardware. They are also well suited to automated research because they have clear metrics, relatively low variance, and evaluators that can be hardened against reward hacks.
We are open-sourcing artifacts from these runs so others can inspect and build on the system’s outputs.
| Benchmark | Task Type | Metric | Previous State of the Art | Recursive | Improvement |
|---|---|---|---|---|---|
| NanoChat Autoresearch | Train a small language model to highest performance given a small compute budget | Validation BPB | 0.9372 | 0.9109 | 0.0263 lower Validation BPB, or a 1.3x speedup to reach the same loss |
| NanoGPT Speedrun | Train a small language model to a certain performance as fast as possible | Training time required to reach a 3.28 validation loss | 79.7 s | 77.5 s | 2.2s faster training |
| SOL-ExecBench | Optimize GPU kernels toward hardware limits | Mean SOL score across 235 kernels | 0.699 | 0.754 | 18% reduction in gap to the optimal performance estimate of 1.0 |
Case study 1: NanoChat Autoresearch
Andrej Karpathy’s NanoChat autoresearch repo is a popular starting point for automated research systems. The task is to train a small language model to the lowest validation loss, measured in bits per byte (BPB), within a fixed five-minute budget on a single GPU.
Perhaps for those reasons, a public collaborative effort has already formed around this setup. autoresearch@home extends the original setup into a collaborative setting where several dozens of humans and hundreds of their agents collectively improve performance. That gives us a stronger comparison point than Karpathy’s single overnight run. We wanted to test if our system could improve on solutions produced by an entire community of humans and agents.
Our system starts from the same initial seed solution the Autoresearch code starts from. After removing minor reward hacks from the previous best autoresearch@home solution and evaluating it on 10 random seeds, its mean performance is 0.9372 BPB. Our system found a solution that reached 0.9109 BPB, a 0.0263 BPB improvement. Measured another way, our solution reaches the quality of Karpathy’s original overnight autoresearch BPB in roughly 1.3x less training time than the best autoresearch@home solution.
We also tested whether our system could make improvements from a much weaker starting point, a naive initial implementation (a vanilla Transformer with AdamW). Our system improved the model from 1.059 BPB to 0.9344 BPB, again outperforming the best solution produced by the autoresearch@home community.
What modifications did our system come up with? The best solutions were not driven by one trick. They combined changes to architecture, short-context memory, auxiliary losses, attention, optimizer behavior, weight decay schedules, compiler settings, and more.
One of the biggest gains came from a richer short-context memory mechanism. The baseline already uses value embeddings; our system extended this idea with hashed bigram and trigram embedding tables, mixed into the attention value path through learned gates. This gave the model a cheap way to use local n-gram information without paying the time cost of slower convolutional or attention-heavy alternatives.
Case study 2: NanoGPT Speedrun
Despite an entire community of humans (sometimes with AI assistance) spending years working on this problem, Recursive’s automated AI research system still discovered additional improvements. Starting from a solution refined by the community over 2+ years, the system still found a sequence of discoveries to produce a further speedup: 79.7s → 77.5s.
Case study 3: SOL-ExecBench GPU kernel optimization
Our system achieved a mean NVIDIA SOL-ExecBench score of 0.754, an 18% reduction in the gap to the hardware limit from the previous leaderboard best of 0.699 across 235 kernels.
These results are an early sign that our system can push the frontier on AI training and infrastructure tasks, especially when the goal is well-defined, measurable, and quick enough to evaluate many times. The system made progress by compounding many discoveries: inventing new optimizations, recasting known ideas under tighter constraints, tuning implementation details that mattered, and composing improvements across modeling, optimization, and systems layers.