ast-grep Rewrites Tree-sitter Core in Rust, Gains 30% Speed with AI Help

ast-grep has rewritten the C core of its Tree-sitter parser in Rust with assistance from OpenAI's Codex. The structural code-search tool achieved a 29.74% speedup in raw parsing throughput and a 22% end-to-end performance gain.

The Numbers

The Rust parser core processes files 29.74% faster than the original C implementation. Tree traversal improved 10.16%. The overall tool runs approximately 22% faster in real-world usage, according to the project's blog.

The Rust core consumes about 8 MiB more memory—a tradeoff the team considered acceptable for the performance gains.

Why Rewrite Tree-sitter?

Tree-sitter powers GitHub's code search and many editor plugins. Its C core is battle-tested but introduces overhead for tools like ast-grep that parse entire files in batch rather than incrementally updating syntax trees as users type.

The ast-grep team removed incremental parsing entirely. This Tree-sitter feature is essential for text editors but unnecessary for batch-processing workflows. Stripping it out allowed the team to optimize for their specific use case.

Generated parsers remain compatible with existing Tree-sitter grammar artifacts, preserving ecosystem interoperability.

AI as Co-Author

The developer used ChatGPT and Codex to write, test, and profile much of the Rust implementation. According to a post on X by the project author, the human set goals, constraints, architecture, and benchmark feedback while the AI handled implementation details.

This workflow demonstrates how AI can accelerate migration of legacy C code to memory-safe languages like Rust, particularly for well-defined refactoring tasks with clear performance targets.

Context

ast-grep converts source code into syntax trees for structural search and refactoring. Because it must parse every file it touches, parser efficiency is the primary bottleneck. The original Tree-sitter C library was not optimized for ast-grep's whole-file batch processing model.

The rewrite shows that targeted optimization—removing unneeded features and leveraging AI for implementation—can yield meaningful speedups even for mature, widely-deployed infrastructure like Tree-sitter.