Postgres Queues Scale to 30,000 Workflows Per Second, But MVCC Bloat Persists
DBOS demonstrates high-throughput scaling for database-backed queues, while critics warn that Postgres' architecture creates long-term performance risks.
DBOS has published a technical guide claiming that Postgres-backed queues can scale to over 30,000 workflows per second. These findings challenge the common industry assumption that dedicated message brokers are required for high-throughput queuing.
To achieve these speeds, DBOS employs a conditional approach to isolation levels. According to the company, using REPEATABLE READ for queues that require global flow control and READ COMMITTED for others allows the system to eliminate serialization failures. This optimization enables the database to handle massive workloads that typically crash or slow down standard relational queue implementations.
The MVCC Bloat Problem
Despite the throughput gains, the technical community has raised concerns regarding the long-term stability of this approach. On Hacker News, developers pointed out that the DBOS guide omits the "bloat" problem inherent to Postgres' Multi-Version Concurrency Control (MVCC) architecture.
In a standard Postgres queue, workers typically use "SELECT ... FOR UPDATE SKIP LOCKED" followed by a DELETE or UPDATE operation to mark a task as complete. Because of MVCC, these operations do not immediately remove data but instead create "dead tuples." Over time, these tuples accumulate, putting immense pressure on the autovacuum process and leading to index bloat. This often results in "performance drift," where a system that works in a toy example degrades under sustained production load.
The Trade-off for Developers
For engineering teams, the debate centers on the trade-off between operational simplicity and long-term reliability. Using an in-database queue provides ACID guarantees and a single source of truth, removing the need to manage a separate cluster like Kafka. However, the risk of performance degradation due to dead tuples has historically pushed high-scale users toward dedicated brokers.
The Path to Zero-Bloat
The emergence of alternative implementations suggests the performance gap can be closed without leaving the database. PgQue, for example, proposes a "zero-bloat" strategy. Rather than relying on per-row deletes that trigger MVCC overhead, PgQue utilizes snapshot-based batching and table rotation via TRUNCATE operations to clear processed tasks efficiently.
Industry observers are now watching to see if these rotation-based strategies can maintain the 30,000+ workflow throughput claimed by DBOS while solving the underlying vacuuming issues. If successful, the need for external message brokers in many architectures may diminish.