French Startup Kog Uses GPU Reverse-Engineering to Boost LLM Speed

French startup Kog is developing a specialized inference engine designed to drastically accelerate large language model (LLM) performance on standard datacenter GPUs. The company is challenging the industry assumption that GPUs are fundamentally ill-suited for the decoding phase of inference, aiming instead to unlock latent power in existing hardware.

The company's Kog Inference Engine (KIE) targets high-end hardware, including the Nvidia H200 and AMD MI300X. To prove the potential of its approach, Kog demonstrated a throughput of 3,000 tokens per second (TPS) per request using the open-sourced Laneformer 2B model. This performance was achieved utilizing a configuration of eight AMD MI300X GPUs, signaling a significant leap in speed for small-scale models.

The Engineering Approach

While much of the AI industry is pivoting toward purpose-built inference chips to solve speed bottlenecks, Kog is betting on low-level software optimization. The company's strategy is led by founder Gaël Delalleau, who applies a non-traditional background in solid-state physics and offensive cybersecurity to the problem. By utilizing white-hat hacking techniques, Delalleau reverse-engineers GPU behavior at the assembly and binary levels to bypass inefficient software abstractions.

According to Delalleau, the belief that GPUs are poorly suited for decoding is a misconception. He argues that newer GPU architectures possess increasing amounts of memory bandwidth that simply remains untapped by current software stacks. By operating at a deeper level of the hardware, KIE seeks to bridge the gap between theoretical hardware capability and actual real-world performance.

Industry Implications

This optimization is particularly critical for the rise of agentic workflows and professional AI tools, such as autonomous software engineering agents, where inference speed and cost are primary bottlenecks. If these gains can be scaled, it could significantly reduce the industry's reliance on expensive, specialized AI silicon and lower the barrier for deploying complex, real-time AI applications on standard infrastructure.

Scaling for the Future

Having established a baseline with the 2B model, Kog is now focusing on scaling these optimizations to larger, more complex models to meet growing customer demand. The company's success will depend on whether the extreme gains seen in small models can be replicated in the massive parameter counts required for frontier-level AI, a transition that remains the primary technical hurdle for the KIE engine.