Moonshot AI Releases 2.8T-Parameter Kimi K3 With 1M Token Context

Moonshot AI has released Kimi K3, a 2.8-trillion-parameter open-weight model that ranks among the largest publicly available AI systems. The architecture supports a 1 million token context window and includes native multimodal capabilities, positioning it as a frontier-scale option for researchers and developers working outside closed API ecosystems.

LatentMoE at Scale

Kimi K3 builds on the Kimi Linear architecture (48B parameters), scaling it up through a Stable LatentMoE framework. The model uses a highly sparse Mixture-of-Experts design, activating only 16 of 896 experts per token. This approach follows a broader industry shift seen in systems like DeepSeek V4 and Nemotron 3, where efficiency-tweaked transformer components handle massive scale without proportional compute cost increases.

The architecture delivers approximately 2.5x improvement in overall scaling efficiency compared to Kimi K2, according to technical documentation hosted on Hugging Face.

Dropping RoPE for NoPE

In a notable architectural departure, Kimi K3 completely omits Rotary Positional Embeddings (RoPE) across all layers. Instead, it uses No Positional Embeddings (NoPE). This design choice aligns with the model's Kimi Delta Attention (KDA) mechanism, which operates with RNN-like efficiency characteristics rather than standard attention patterns.

The model also implements Attention Residuals (AttnRes), a modification that improves validation loss during training. According to Sebastian Raschka's architecture analysis citing the technical report, AttnRes adds approximately 4% to training cost and 2% to inference cost.

Efficiency Mechanisms

KDA and AttnRes work together to optimize both training and inference performance at scale. The combination allows Kimi K3 to maintain high capacity while managing the computational demands typical of trillion-parameter systems. The vLLM team has already published preview support for production-scale deployment, signaling early infrastructure readiness.

Open-Weight Implications

The release of a 2.8T parameter model with open weights significantly lowers barriers for researchers studying ultra-large-scale MoE architectures. Developers can now experiment with million-token context windows and native vision capabilities without relying on proprietary API access.

Kimi K3's architecture suggests a potential shift in how frontier models handle position encoding and attention mechanisms. The move away from RoPE toward NoPE, paired with KDA's RNN-like behavior, may influence future designs in both open and closed model development.

The model is available on Hugging Face under the moonshotai organization.