Principles for Fast Tokio Applications

When writing performant Tokio applications, there are no strict rules, as the answer to many questions depends on the specific workload and runtime environment. Performance is a balance between fairness and batching, contention and isolation. This article outlines general principles, with exceptions where applicable, assuming familiarity with Tokio's work-stealing runtime. High-level summaries are provided in the appendix.

One common issue is long polls (time between .await points when code yields back to the runtime) exceeding 10-100 microseconds. While these may or may not impact application metrics, they should be addressed based on the specific performance goals. It's crucial to work backward from a real metric you aim to improve.

In most cases, performance problems stem from application code itself, often due to interactions between multiple components in a distributed system, rather than the runtime itself. dial9 can help identify these issues, but equally often it demonstrates the absence of a Tokio problem, encouraging developers to look elsewhere.

The most useful Tokio metric is schedule latency histogram, which measures the time between a task being ready to run and Tokio polling the future. While this doesn't pinpoint the cause, low schedule latency indicates fair interaction between Tokio and your code. Fairness is essential for high throughput, as seen in Redis implementations. Explicitly yielding after each request can reduce latency by up to 10×. Additionally, yielding between several consecutive immediately-ready reads can further improve latency.

Tokio's efficiency is directly related to how much useful work you can perform per runtime event. Tools like tokio::fs can help optimize performance, but it's essential to batch or thread filesystem operations to minimize overhead. Spawning tasks can also introduce overhead, so consider batching tasks to reduce the impact of the runtime's work-stealing scheduler. Tools like dial9 or tokio-metrics can help track task lifecycles and identify potential bottlenecks.

The Tokio runtime schedules work on workers, which are dedicated threads that poll ready tasks. Workers scale across cores, but global resources, like the blocking pool, can become bottlenecks at high rates. When push work onto the blocking queue, it can lead to negative performance effects, as seen with spawn_blocking at roughly 50,000 blocking tasks per second on a 32-core host.

It's crucial to evaluate the specific use case and determine whether Tokio's workers and work-stealing capabilities are sufficient, or if alternative approaches, like letting the OS handle blocking or CPU-heavy work, are more appropriate.