PicoMQ Launches Rust-Based Stream Server Using S3 for Durable Storage

PicoMQ has introduced a durable stream server written in Rust that offloads data persistence to S3-compatible object storage. The system allows developers to create millions of granular, URL-addressable streams that incur no cost while idle.

According to official documentation, PicoMQ utilizes S3-compatible object storage for all records, including the write-ahead log. This design ensures that data durability remains independent of the compute nodes. For cluster coordination, the system employs an ordered command log stored in a SQL database—using Postgres for clusters and SQLite for single-node deployments—which removes the requirement for a custom consensus protocol. The server supports two wire protocols: the proprietary Pico protocol and the open Durable Streams protocol, both of which operate over standard HTTP using PUT, POST, GET, and Server-Sent Events (SSE).

The Shift to Stateless Infrastructure

Traditional message brokers typically rely on complex partition management and dedicated disk arrays to maintain performance and durability. This often creates operational bottlenecks during scaling, as adding or replacing nodes requires time-consuming data rebalancing. PicoMQ addresses this by implementing "zero-disk" nodes. Because these nodes are stateless and hold no unique data, they can be replaced or scaled rapidly without moving records between servers. This architecture shifts the storage burden to object storage, which is generally more scalable and cost-effective than high-performance broker disks.

Enabling Stream-per-Entity Architectures

This architectural shift enables a "stream-per-entity" model, where a unique stream can be assigned to every individual user session or device. In traditional brokers, managing millions of such granular streams would be prohibitively expensive and operationally complex. By making streams disposable and cheap, PicoMQ facilitates new patterns for real-time event history, agent conversations, and massive-scale audit trails without the overhead of managing consensus clusters.

Performance Trade-offs and Outlook

While the system offers high availability and scalability, the reliance on object storage introduces specific latency characteristics. Documentation notes that an append operation typically costs tens of milliseconds due to the round trip to object storage, indicating the system is not intended for use cases requiring single-digit millisecond latency. Future adoption will likely depend on whether the operational simplicity of stateless nodes outweighs the latency costs for high-frequency streaming applications.