Definition
- 2026-08-03: qwen3-8-max ships as 2.4T MoE with ~95B active (vendor) (2026-08-03-alibaba-qwen38-max-alicloud-blog)
Mixture of Experts (MoE) is a neural network architecture where only a subset of model parameters (“experts”) activate per inference token, enabling very large total parameter counts with manageable compute per request. Sparse MoE designs route each token to a small number of expert layers.
Key Characteristics
- 2026-07-31: deepseek-v4-flash 0731 keeps 284B/13B MoE architecture; gains from re-post-training not new MoE layout (2026-08-01-deepseek-v4-flash-0731-official-release)
- Total vs active parameters: Large total count (e.g., 310B) with small active subset (e.g., 15B) per forward pass
- Efficiency: Lower inference cost than dense models of equivalent capability
- Scaling: Enables frontier-class models without proportional compute increase
MiMo-V2.5 (Xiaomi, April 2026)
- MiMo-V2.5: 310B total parameters, 15B active, 1M token context, natively multimodal
- MiMo-V2.5-Pro: ~1.02T parameters, 42B active, hybrid-attention architecture
- MIT-licensed; pricing from $0.40/M input tokens (among cheapest frontier models)
- Powers mimo-code MiMo Auto channel (free limited-time)
- Post-training includes “harness awareness” — model trained to manage memory within agent scaffolds