Turso Research: O_DIRECT in io_uring Creates Critical I/O Bottleneck

Developers utilizing io_uring with O_DIRECT may be inadvertently crippling storage performance by disabling critical kernel optimizations. New research from Fernando Simões reveals that bypassing the page cache removes implicit kernel readahead, forcing applications to implement their own prefetching to maintain I/O concurrency.

While analyzing the io_uring backend for the Turso database, Simões found that O_DIRECT creates a bottleneck where the application submits a single request and waits for its completion before submitting the next. This sequential behavior prevents the kernel from seeing multiple pending requests, which is a prerequisite for I/O merging at the block layer. To solve this, Turso implemented an application-level readahead window of 32 pages.

The results were dramatic: for a TPC-H Q6 query, the number of device requests plummeted from approximately 196,000 to just 16,300. Furthermore, the percentage of read requests merged (%rrqm) surged from nearly 0% to between 91% and 93%.

The Mechanics of I/O Merging

This performance gain stems from how the Linux block layer handles contiguous data. If two requests in the queue cover adjacent sectors, the block layer can join them into a single, larger request. However, this optimization only functions if both requests reside in the queue simultaneously. By prefetching pages at the application level, Turso ensures the queue remains populated, allowing the kernel to merge adjacent requests and significantly reduce the total load on the NVMe device.

Hardware and CPU Trade-offs

Beyond request counts, the research explored the architectural impact of O_DIRECT. Because O_DIRECT avoids copying data from the page cache to the process buffer, it alters the CPU's cache behavior. Tests indicated that this approach results in higher CPU cache misses compared to buffered syscalls, with miss rates rising from 7.691% to 13.255% during queries.

Additionally, the study examined the use of Submission Queue (SQ) polling. Simões noted that removing the SQ polling thread slightly increased wall time but significantly reduced system time. This suggests that sqpoll is most effective when there are free vCPUs available, preventing the polling thread from competing for resources with the primary query thread.

Implications for Database Engineering

This research highlights a significant pitfall for engineers seeking the raw performance of io_uring. While O_DIRECT is often used to reduce CPU overhead and avoid double-buffering, the loss of implicit kernel readahead can lead to orders-of-magnitude increases in device-level requests for sequential scans. To saturate the capabilities of modern NVMe drives, developers must shift the responsibility of prefetching from the kernel to the application layer.

Moving forward, the industry must weigh the benefits of reduced CPU copies against the necessity of manual I/O orchestration. As databases move toward more direct hardware interaction, the burden of optimizing data flow increasingly falls on the application developer rather than the operating system.