This page may contain stale information. Last updated: 2026-04-24
Definition
Distributed AI training refers to training machine learning models across multiple computing devices (chips, servers, data centers) working in coordination. This approach is essential for frontier models requiring massive compute resources.
Key Challenge: Fault Tolerance
Training across thousands of chips faces:
- Hardware failures (chip, network, power)
- Bandwidth constraints between sites
- Synchronization overhead
DiLoCo Framework
Google DeepMind’s decoupled DiLoCo addresses these challenges:
- Maintains 88% goodput across thousands of chips
- Significant bandwidth reduction (from ~198 Gbps to much lower)
- Resilient to hardware failures
- Enables cross-site training without perfect networks
Economic Impact
- Training costs: 100M+ for frontier models
- One chip failure in thousands-chip cluster can waste weeks
- Efficiency improvements have massive economic impact