The race to build bigger, better AI models is on. But does simply increasing size lead to improved performance? Not always. A fascinating new research paper from Apple, "Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training," challenges the conventional wisdom about scaling large language models (LLMs). Traditionally, researchers have focused on metrics like FLOPs (floating-point operations) to measure model complexity. However, this overlooks the communication overhead in Mixture-of-Experts (MoE) models, which use a collection of specialized “experts” to handle different parts of a problem. This research takes a fresh look at the problem, using *step time* as a more accurate measure of complexity. The results are surprising. By optimizing MoE models with a clever 3D sharding strategy, the researchers demonstrate that MoEs can outperform denser models, even when using the same compute budget. This means faster training and better performance, opening doors to more efficient and powerful AI. The team tested their models on a variety of tasks, from common sense reasoning to complex mathematical problems. Across the board, MoEs consistently came out on top. This research has significant implications for the future of AI. By focusing on efficient architectures like MoE, we can unlock even greater potential from these powerful tools, paving the way for more sophisticated and capable AI systems.