Moonshot AI Open-Sources MoonEP for Distributed MoE Communication

After optimizing parallelism strategies to minimize communication volume, we further reduce the communication overhead to nearly zero using comprehensive communication-computation overlapping techniques. Training large models involves integrating various techniques, which increases the complexity of communication overlap. For instance, at any given moment, the device might concurrently handle computation and communication kernels, overlap PP and DP communications, and manage data transfers between the device and host. [...] We adopt sequence parallelism (SP), as proposed in DeepSpeed-Ulysses , to scale MoE training and effectively reduce communication along the critical path. SP is commonly used in long-context training to address memory challenges associated with long inputs. We find it also works well in large-scale MoE training. First, it significantly reduces communication overhead compared to TP, especially when using grouped-query attention . Second, while it introduces some para

open-source is beautiful, indeed We've open-sourced MoonEP, our high-performance communication library for distributed MoE workloads. Built to make expert-parallel communication more efficient at scale, MoonEP helps reduce communication overhead in large MoE training and inference systems. Explore on GitHub: ## Combined views 177.6K 8 posts, first seen 5d ago ## Sentiment ### Positive Read Positive users praise MoonshotAI for open-sourcing MoonEP because it tackles communication overhead and efficiency limits in large-scale MoE training, while the lone negative reply dismisses AMD hardware mentions. Based on 14 visible X reactions from 86 accounts; directional sample. ## Useful Links GitHub ## Digg Deeper Ask a question below. Published answers will appear here. [...] Very interesting >cutedsl kernels We've open-sourced MoonEP, our high-performance communication library for distributed MoE workloads. Built to make expert-parallel communication more efficient at scale

Zero copy makes raw communication faster: tokens are written directly to their final expert-grouped positions on remote ranks — no permute in, no permute out — and views of the communication buffer are handed straight to the computation, eliminating the comm-buffer → user-buffer copy that dominates the epilogue. MoonEP's comm time is consistently below DeepEP v2 at every imbalance level. Perfect balance makes it immune to imbalance: MoonEP's comm time stays almost flat as maxvio grows, while DeepEP v2 — whose latency is set by the hottest rank — degrades steadily. [...] 1. Perfect balance: every rank receives exactly `S × K` tokens, no matter how skewed the routing is. A small number of redundant experts is planned online from the current router outputs and prefetched before expert computation; their gradients are reduced back to their home ranks in the backward pass. 2. Online planning: a near-optimal GPU planning kernel with negligible overhead 3. Zero copy and static shapes: fused

The library is fully open-source and available at . In this article, we explore the challenges of expert parallelism in large-scale MoE models and describe our approach to efficient token dispatch and combination across distributed GPU environments. ### Introduction [...] # Efficient and Portable Mixture-of-Experts Communication An overview of portable Mixture-of-Experts (MoE) communication, focusing on optimizing GPU parallelism and reducing latency in large-scale AI models We present a high-performance, portable, open-source library for Mixture-of-Experts (MoE) communication that achieves 10x faster performance compared to standard All-to-All communication primitives. Our implementation features several key technical innovations that deliver superior MoE communication efficiency: [...] Our approach balances performance with portability through key innovations including GPU-initiated communication support, a split kernel architecture enabling computation-communication overlap, an

# Moonshot AI Open-Sources MoonEP: A Perfectly Balanced Expert Parallelism Library for MoE Training By Michal Sutter - Moonshot AI has open-sourced MoonEP, an Expert Parallelism (EP) communication library for distributed Mixture-of-Experts (MoE) workloads. The team announced the release as a library built to make expert-parallel communication more efficient at scale. It ships under an MIT license. [...] MoonEP arrived as part of Kimi K3 Open Day. Alongside the K3 model weights and technical report, Moonshot released three infrastructure codebases: MoonEP, FlashKDA, and AgentEnv. FlashKDA had already been open-sourced; MoonEP and AgentEnv were published with this release. MoonEP is one of the innovations behind a claimed 2.5× improvement in scaling efficiency for Kimi K3, a 2.8-trillion-parameter MoE model with native vision and a 1M-token context window. ## The problem MoonEP targets In expert parallelism, a router sends each token to its top-K experts, which live on different ra

# Moonshot AI Open-Sources MoonEP: A Perfectly Balanced Expert Parallelism Library for MoE Training By Michal Sutter - Moonshot AI has open-sourced MoonEP, an Expert Parallelism (EP) communication library for distributed Mixture-of-Experts (MoE) workloads. The team announced the release as a library built to make expert-parallel communication more efficient at scale. It ships under an MIT license. [...] MoonEP arrived as part of Kimi K3 Open Day. Alongside the K3 model weights and technical report, Moonshot released three infrastructure codebases: MoonEP, FlashKDA, and AgentEnv. FlashKDA had already been open-sourced; MoonEP and AgentEnv were published with this release. MoonEP is one of the innovations behind a claimed 2.5× improvement in scaling efficiency for Kimi K3, a 2.8-trillion-parameter MoE model with native vision and a 1M-token context window. ## The problem MoonEP targets In expert parallelism, a router sends each token to its top-K experts, which live on different ra