Amazon AGI Researchers Warn FLOP-Based Scaling Laws Mislead MoE Training

Researchers from the Amazon AGI Foundation have introduced MOSAIC, a framework that challenges the industry's reliance on traditional scaling laws for training large language models. The team discovered a critical disconnect between theoretical compute-optimality and the actual GPU-hours required to train sparse Mixture-of-Experts (MoE) models.

Traditional scaling laws optimize for Model FLOPs, but hardware clusters bill in GPU-hours. This gap means that an architecture appearing efficient on paper may be prohibitively expensive or slow in practice. As researcher Sheng Zha noted, "Clusters do not bill FLOPs. They bill GPU-hours."

To bridge this gap, the team developed the Model Optimization via Systems-Aware Training Co-design (MOSAIC) framework. MOSAIC couples a predictive scaling law with a calibrated performance model to estimate Model FLOPs Utilization (MFU), memory footprint, and communication costs. This allows developers to integrate model architecture selection directly with systems-level performance modeling.

The Sparsity Paradox

The research highlights a specific issue with sparse MoE models. In budgets that ignore efficiency and focus solely on model-FLOPs, these models show no interior optimal sparsity. Instead, loss decreases monotonically as models become sparser, which pushes the theoretical "optimum" to the extreme boundary of data support. This suggests that without accounting for hardware constraints, scaling laws would push architects toward extreme sparsity that may not be viable on actual clusters.

To validate these findings, the researchers based their scaling law fits on roughly 150 from-scratch sparse MoE pretraining runs. These tests involved total model sizes reaching up to 79 billion parameters, with active parameters ranging from 104 million to 2.7 billion.

Why Cluster-Optimality Matters

This shift from compute-optimal to cluster-optimal design is vital because different architectures utilize hardware with varying efficiency. Two models with identical FLOP counts can have vastly different training times and costs depending on how they handle communication and memory. For complex architectures like MoEs, the optimal sparsity is not a fixed property of the model but depends heavily on the specific hardware cluster's constraints.

By adopting a systems-aware approach, developers can avoid choosing architectures that look efficient in theory but are bottlenecked by communication overhead on real-world hardware. This transition is expected to lead to faster and more cost-effective training for frontier models.

What's Next

The introduction of MOSAIC provides a blueprint for moving beyond FLOP-centric design. The industry must now determine how to standardize these performance models across different hardware generations. While the research proves that systems-aware scaling is necessary for MoEs, the extent to which this applies to other emerging architectures remains a key area for further investigation.