Akamai and Neural Magic Shift AI Inferencing to CPU-Based Edge Network
The partnership leverages model sparsification to run deep learning on commodity hardware, bypassing the need for expensive GPUs.
Akamai Technologies has partnered with Neural Magic to enable efficient AI inferencing on CPU-based servers across its distributed edge network. The collaboration aims to lower the barrier for deploying data-intensive AI applications by reducing the industry's reliance on scarce and costly GPU resources.
To achieve this, the two companies are integrating automated model sparsification software. This technology allows complex deep learning models to run on commodity CPU hardware rather than requiring specialized accelerators. By shifting these workloads to the edge, companies can process data closer to the end user, which significantly reduces latency and improves overall cost-efficiency.
The GPU Bottleneck
Traditionally, AI inferencing—the phase where a trained model makes real-time predictions—has required powerful GPUs. While effective, these chips are expensive and difficult to deploy at scale within remote edge locations due to high power requirements and delivery constraints. Edge computing is designed to process data near its source to save bandwidth, but the hardware demands of deep learning have historically made this impractical for many organizations.
John O’Hara, senior vice president of engineering and COO at Neural Magic, noted that specialized hardware and its associated power requirements are not always feasible, which has previously left organizations unable to leverage the benefits of running AI inference at the edge.
Scaling Global AI
By moving inferencing from centralized GPU clusters to distributed CPU-based nodes, organizations can scale AI workloads globally without the prohibitive costs of GPU infrastructure. This shift democratizes access to high-performance AI for real-time applications and allows for more complex data processing at the point of entry.
According to Ramanath Iyer, chief strategist at Akamai, the move allows customers with large deep learning models to utilize a cost-efficient, CPU-based platform to deploy their applications at scale with improved performance.
Future Implications
As AI models grow in size and complexity, the ability to run them on standard server hardware becomes a critical competitive advantage. The industry is now watching to see how this sparsification approach handles the most demanding generative AI models and whether it can fully replace the need for edge-based GPUs in high-throughput environments. While the technical foundation is set, the primary focus remains on how many enterprises will migrate their existing centralized models to this distributed CPU architecture to realize the promised latency gains.