Qwen3.8-27B Hits 50 Tokens Per Second on 24GB RTX PRO 4000 SFF
Michał Piszczek demonstrates that precision tuning and speculative decoding enable high-throughput, large-context inference on professional SFF hardware.
Local AI deployment has reached a new efficiency milestone as the Qwen3.8-27B model was optimized to run at high speeds on a single professional GPU. This development proves that large-parameter models can maintain massive context windows without sacrificing real-time performance on small form factor (SFF) hardware.
Michał Piszczek achieved an average throughput of 50.44 tokens per second (tok/s) using an NVIDIA RTX PRO 4000 Blackwell SFF GPU equipped with 24GB of VRAM. The setup successfully supports the model's full context window of 262,144 tokens. To reach these speeds, Piszczek utilized Multi-Token Prediction (MTP) as a speculative decoding mechanism, which allows the system to predict multiple subsequent tokens simultaneously rather than one by one.
The Precision Trade-off
The experiment highlighted a critical relationship between memory allocation and speed. Piszczek discovered that the precision of the 'drafter'—the component responsible for speculative predictions—is a primary bottleneck for performance. In one test, increasing the MTP drafter precision by just 69.2 MiB caused throughput to plummet from 50.44 tok/s to 37.02 tok/s.
This result underscores a broader challenge in local LLM deployment: the tension between theoretical precision and actual throughput. Qwen3, released by Alibaba under the Apache 2.0 license, features dense and MoE architectures designed for reasoning and coding. While the 27B dense model is powerful, running it with a 256K+ context window typically demands immense VRAM, making these optimization breakthroughs essential for professional SFF environments.
Industry Implications
This deployment demonstrates that high-throughput inference for 27B-parameter models is viable on a single 24GB professional GPU, provided the software stack is tuned. It suggests that the industry should shift focus from simply adding more VRAM to optimizing the interaction between the main model and its speculative drafters. As Piszczek noted, "the best local inference setup is rarely made from the individually 'best' parts."
Future Outlook
The success of this configuration points toward a future where professional workstations can handle enterprise-grade reasoning tasks locally without relying on cloud clusters. The primary remaining challenge for the community is identifying the optimal precision floor for various drafter architectures to maximize speed without degrading output quality. For now, the RTX PRO 4000 Blackwell SFF stands as a proven platform for high-efficiency, large-context local AI.