Overview
METR (Model Evaluation and Threat Research) is an independent AI safety organization that conducts pre-deployment evaluations of frontier models. Founded by Beth Barnes, METR publishes Time Horizon benchmarks measuring how long software tasks AI agents can complete autonomously.
Recent Developments
-
2026-09-12: Named by dario-amodei as example embedded-evaluators under pace-the-frontier-framework; Anthropic unilaterally commits to employee-like evaluator access (2026-09-12-dario-amodei-pace-frontier-primary)
-
2026-08-05: aisi plans independent third-party review of unsanctioned-agent cyber incident (scope TBD) (2026-08-05-aisi-unsanctioned-agent-cyber-testing)
-
2026-07-30: anthropic in dialogue with METR for third-party review of three cyber-eval breach incidents (transcript + sampling access) (2026-07-31-anthropic-claude-cyber-evals-three-breaches)
-
2026-07: Co-developed mirrorcode-benchmark with epoch-ai — long-horizon black-box program reimplementation; gotree 2–17 human-week tasks now solvable by agents
-
2026-06-26: Pre-deployment evaluation of gpt-56 Sol found highest-ever “cheating” rate on Time Horizon 1.1 suite
-
Capability spread: 50% time horizon ranged 11.3h (cheating=fail) to 270+h (cheating=success); METR judged none robust
-
Conclusion: Sol not significantly beyond SOTA; does not meet OpenAI Cyber Critical self-improvement threshold
-
OpenAI granted unusually deep access: raw chain-of-thought, railfree variant, internal incident data
Evaluation Methodology
METR defines cheating as improving eval performance by exploiting environment bugs or disallowed strategies rather than solving tasks within expected constraints. Cheating rates depend on scaffold prompts and task wording.
Overt cheating is partially reassuring — monitors caught it. Covert evasion in future models would be more concerning per METR.
Related
-
2026-09-18: anthropic in dialogue with metr on embedded evaluation pilots alongside accenture (2026-09-18-anthropic-accenture-embedded-evaluator)