New Verifier Finds 40% of 'Correct' LLM-Generated GPU Kernels Are Broken

Researchers Rishi Shah and Rishav Shrestha have developed a "contract-grade" verifier designed to audit GPU kernels generated by large language models (LLMs). The tool reveals that a significant portion of machine-generated kernels, previously accepted as correct by industry-standard benchmarks, contain critical failures.

According to the researchers, the verifier audited 2,638 machine-generated kernels that had already been passed by a public system's own testing harness. The results showed that 39.5% of these kernels were broken beyond any tolerance argument, and 62.1% carried at least one violation. In a direct comparison, the standard industry test accepted 1,487 kernels that the new verifier rejected, while the standard test rejected only 14 that the verifier deemed acceptable.

The Failure of Loose Testing

As LLMs are increasingly used to automate the creation of high-performance GPU kernels, the industry has largely relied on "loose" testing. This traditional approach typically involves running kernels on a small set of random inputs at a single shape and checking if the output is "close enough" to a reference value.

Shah and Shrestha argue that this methodology is too lenient and fails to catch silent failures. Specifically, loose tests often miss non-deterministic results, failures that occur when input shapes change, or the incorrect handling of infinities and NaNs (Not-a-Number). These bugs can remain hidden during initial testing but cause catastrophic failures in production environments.

A Contract-Based Approach

To solve this, the new verifier employs twelve adversarial gates to stress-test the kernels. Some of these gates are "tolerance-free," meaning that any failure detected cannot be explained away or dismissed by simply adjusting numerical thresholds. This rigorous framework ensures that the kernel adheres to a strict contract of correctness rather than just passing a superficial check.

The authors applied this system to their own work, using the verifier to judge a native Blackwell tcgen05 training backward for the gated-linear-recurrence (GDN) family. By testing it against a double-precision oracle, they were able to establish the kernel's actual correctness.

Implications for AI Infrastructure

This research suggests that the reported progress in LLM-based kernel generation may be inflated due to inadequate testing protocols. When kernels are deployed in critical AI infrastructure, silent failures can lead to unstable training runs or incorrect model outputs that are difficult to debug.

By introducing a contract-based verification system, the researchers provide a pathway to ensure that generated kernels are truly reliable for production use. The shift toward adversarial auditing suggests that the industry must move away from probabilistic "closeness" and toward deterministic verification to safely integrate LLM-generated code into the hardware stack.

What's Next

As the industry moves toward more specialized hardware like the Blackwell architecture, the need for rigorous verification will likely grow. It remains to be seen if public kernel-generation benchmarks will adopt these adversarial gates to provide more honest assessments of LLM capabilities.