Tokio's Work-Stealing Scheduler Can Spike Memory at 1M Tasks

The Hidden Cost of Massive Task Fan-Out

A Rust developer building a high-throughput event service discovered that spawning up to one million Tokio tasks in a fan-out pattern triggered unexpected memory spikes. Tokio's work-stealing scheduler prioritizes progress over ordering, leaving parent tasks alive far longer than anticipated.

The developer processed events from a message queue, spawning one Tokio task per event, then fanning out into up to 1,000 sub-tasks per event for user tokens. During a burst of 1,000 events with 1,000 tokens each, memory usage climbed as parent event tasks remained alive waiting for their children to complete.

How Tokio's Scheduler Works

Tokio's multi-threaded runtime assigns each worker thread a local queue with capacity for 256 tasks, plus a shared global queue. Workers primarily pull from their own local queue, check the global queue periodically, and steal from other workers' queues when idle. When a local queue overflows, the worker moves half of its tasks to the global queue.

This architecture ensures all tasks eventually make progress. But it does not guarantee that tasks spawned earlier will be polled or completed earlier.

"Spawned early doesn't mean first-polled early. First-polled early doesn't mean completed early," wrote Pranitha, who documented the investigation.

Why Memory Spiked

During the burst, tasks from earlier events were started much later than tasks from newer events. A few token tasks from early events stayed alive until the end of the burst, keeping their parent event tasks—and all associated event state—in memory.

Parent tasks awaiting JoinSet completion remain alive while their child tasks are pending. With no bound on how many events could be processed concurrently, the application accumulated live task groups faster than they drained.

"If there is an application-level unit of fairness you expect Tokio to honor, like the event in our case, the runtime doesn't know it exists. The bounds should be added by the application," Pranitha noted.

The Fix: Bound Concurrency Explicitly

The solution was to add a Semaphore limiting how many events could be processed concurrently. This reduced peak memory usage without impacting throughput, since the runtime was already saturated with work.

The case demonstrates a critical distinction in asynchronous runtimes: task creation does not equal task polling or completion. Developers assuming FIFO-like behavior for spawned tasks may encounter mysterious memory leaks when scaling to millions of tasks, as the runtime prioritizes work-stealing and progress over strict submission ordering.

For services spawning massive task counts, explicit concurrency bounds at the application level remain essential—even when the runtime guarantees every task will eventually run.