JW Labs Slashes LLM Latency by 82% With Speculative KV Cache Replication
The 'Bite the Bullet' system replaces expensive prefill computations with rapid RDMA transfers during request bursts.
JW Labs has developed 'Bite the Bullet,' a system designed to slash Time to First Token (TTFT) during bursts of LLM requests that share the same prefix. By speculatively replicating Key-Value (KV) caches across GPU replicas, the system eliminates the need for redundant computations on target nodes.
To achieve this, the system identifies synchronized bursts of requests and employs an 'early_rdma' policy to transfer KV caches from HBM, RAM, RDMA, or disk to warm up less-busy GPU replicas. This approach exploits a massive efficiency gap: JW Labs found that prefill computation is approximately 44x more expensive than an RDMA transfer. By moving the data instead of re-calculating it, the system ensures that subsequent requests arriving at a replica can begin generating tokens immediately.
The Prefill Bottleneck
In standard LLM serving, the KV cache allows requests with identical prefixes to reuse previously computed math. However, when a burst of such requests is distributed across different GPU clusters, each cluster typically must perform a 'prefill' phase to compute the prefix. While existing routers like SGLang Model Gateway or Dynamo distribute the load, they do not proactively move the cache to anticipate these burst patterns, leaving the compute-heavy prefill as a primary source of latency.
Performance and Impact
Reducing TTFT is critical for the perceived responsiveness of real-time AI applications. 'Bite the Bullet' transforms a compute-heavy task into a memory-transfer task, allowing clusters to handle prefix-heavy workloads without requiring all requests to land on the same physical node. In tests across various models—including 70B, GLM, Qwen, Kimi, and dense-1T—the system demonstrated mean TTFT improvements ranging from 10% to 60%. Most notably, p95 latency improvements reached up to 82% when compared to the SGLang default router.
Deployment Strategy
To maintain system stability, 'Bite the Bullet' is designed to remain inert during ordinary traffic. It only activates when a sustained shared-prefix burst is detected, ensuring that RDMA resources are not wasted on sporadic or unique requests. The project is currently available via JW Labs' GitHub repository for further evaluation.