Swiftlet Runtime Brings 80B Parameter Models to Consumer Macs and iPhones

Developer leonickson1 has released Swiftlet, a Swift and Metal-based runtime that allows massive Mixture-of-Experts (MoE) models to run on consumer Apple hardware with minimal RAM. By streaming routed expert weights from the SSD on demand, the system enables high-parameter models to operate on devices that would typically lack the memory capacity to load them.

According to the project's GitHub repository, the Qwen3-Next-80B-A3B (4-bit) model can run on an M5 Mac with a peak RAM usage of just 4.3 GB, achieving decode speeds between 4.5 and 5 tokens per second. On mobile hardware, the Qwen3.6-35B-A3B (4-bit) model runs on an iPhone 17 using approximately 2.5 GB of RAM at a speed of about 1 token per second. The project is open-source and has been integrated into the 'Priv AI' app available on the iOS App Store.

The Mechanics of Expert-Streaming

Traditional LLM execution requires the entire model to be loaded into RAM or VRAM, which makes 80B parameter models inaccessible to most consumer devices. Swiftlet addresses this by targeting the Qwen hybrid MoE architecture, where only a small fraction of parameters—approximately 3B—are active per token.

To optimize this process, Swiftlet utilizes a custom '.qpack' container format. This format repacks experts into fixed-stride blobs, allowing the system to use single-read (pread) fetching from the SSD. This specific approach avoids the mmap and page-cache thrashing that often plagues traditional memory-mapping methods. Additionally, the runtime employs Gated DeltaNet linear attention for 75% of its layers, a design choice that prevents the KV cache from growing regardless of the context length.

Shifting the Hardware Bottleneck

This architectural shift moves the primary hardware bottleneck from RAM capacity to SSD read speed. By keeping only a small dense core in memory and fetching experts as needed, Swiftlet drastically lowers the barrier for running state-of-the-art large models locally.

While this method may introduce concerns regarding increased SSD wear and slower prefill times, it demonstrates a viable path toward running trillion-parameter models on consumer-grade hardware. This capability could eventually enable high-reasoning AI tasks on mobile devices without the need for cloud reliance, preserving user privacy and reducing latency for specific operations.

Future Outlook

As the project evolves, the industry will be watching how this expert-streaming thesis scales with even larger model architectures. While the current implementation focuses on the Qwen family, the underlying principle of on-demand weight streaming suggests a future where the physical RAM of a device no longer dictates the maximum size of the AI it can execute. Whether this approach can maintain acceptable speeds for larger contexts remains a key area for further observation.