French Startup Kog Targets 30x Inference Boost via GPU Reverse-Engineering

French startup Kog is developing a specialized inference engine (KIE) designed to drastically accelerate AI inference on existing datacenter GPUs. The company aims to achieve up to 30x faster large language model (LLM) inference, targeting the high-performance requirements of professional AI workflows and real-time agentic applications.

To reach these speeds, Kog targets high-end hardware such as the Nvidia H200 and AMD MI300X. In a recent demonstration, the company achieved 3,000 tokens per second (TPS) for single-request decoding using the Laneformer 2B model running on eight AMD MI300X GPUs. This performance is driven by a new Transformer architecture called Delayed Tensor Parallelism (DTP), which speeds up batch-size-one inference by hiding communication overhead behind computation and weight streaming. The company's methodology involves extreme low-level software engineering, with teams spending weeks or months reverse-engineering a single chip to optimize performance at the assembly and binary levels.

The Battle for Memory Bandwidth

The push for optimization comes as AI inference becomes a primary bottleneck for the next generation of AI, specifically for voice assistants and complex agentic workflows that require near-instantaneous responses. Currently, the industry is divided between two philosophies: building purpose-built AI chips from the ground up or squeezing more performance out of general-purpose GPUs.

Kog is betting on the latter. CEO Gaël Delalleau argues that the notion that GPUs are ill-suited for decoding is a misconception, asserting that newer GPUs possess significant memory bandwidth that simply needs to be unlocked. By operating at the binary level, Kog intends to bypass the inefficiencies of standard software stacks to access this latent power.

Implications for AI Infrastructure

If Kog can successfully scale these gains from small 2B-parameter models to massive, industry-standard LLMs, the economic and technical implications would be substantial. A 30x increase in speed would drastically reduce the latency and operational costs of AI services. For enterprises, this means the ability to deploy sophisticated, multi-step agentic workflows without the massive capital expenditure required to replace existing GPU infrastructure with specialized AI accelerators.

Scaling and Support

Kog is currently backed by Bpifrance and the French Tech 2030 program, with additional support from Scaleway. The company's immediate focus is on demonstrating that these speed-ups are viable for larger models to prove commercial traction. As the industry moves toward more autonomous agents that must "think" and respond in real-time, the ability to maximize existing hardware may provide a faster path to deployment than waiting for the next generation of custom silicon.