This page may contain stale information. Last updated: 2026-07-05
Definition
Test-time scaling refers to increasing compute expenditure during inference — through multi-step reasoning, tool calls, reflection loops, or parallel sampling — to improve output quality at the cost of latency and energy.
Energy Cost (KAIST 2026)
Paper title: “The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective” (IEEE HPCA 2026).
- Reflexion on Llama-3.1-Instruct 70B: 348.41 Wh/query (136.5× single-turn baseline)
- LATS on 70B: 158.48 Wh (62.1× baseline)
- GPU underutilization during serialized reasoning steps amplifies waste
Design Tradeoff
Developers choosing agentic architectures must budget for inference scaling — not just model capability benchmarks. Agent efficiency becomes infrastructure planning constraint as adoption accelerates.
Related
- agent-energy-consumption
- energy-efficiency-in-ai
- ai-infrastructure
- long-horizon-agents
- agentic-ai
- kaist