Tokio's Biggest Performance Killer Isn't Tokio
Principles for Fast Tokio Applications
At RustConf's Unconf, Russell distilled hard-won lessons from debugging async applications into a living document of best practices. The core insight: most performance problems blamed on Tokio actually live in application code — especially in how components of a distributed system interact. He covers when to yield for latency versus batch for throughput, why global resources like the blocking pool become bottlenecks, how a single contended mutex can stall every runtime worker, and why you should pin Tokio workers away from other threads.
In the overwhelming majority of problems I have come across, the issue was in the application code itself, often in the interaction between multiple components of a distributed system (and not actually in Tokio).
- saghm
"Be careful with mutexes" is good advice, but I'm surprised it doesn't explicitly call out the various channels that tokio provides as alternatives (detailed here: https://docs.rs/tokio/latest/tokio/sync/index.html). There are a variety of options that fit different use cases, and you don't even need to enable the runtime feature to use them (e.g. if you want to do a single check for completion rather than await). I'd estimate that at least half of the bottlenecks I've seen with mutexes when using tokio could have been avoided by not even using a mutex at all and instead passing the data that's truly needed across different tasks with some type of channel.
The other trick I've used a few times that's a bit hacky but can get the job done is when reading a snapshot of the data under a mutex is enough without needing to prevent other changes; if that's the case, you can just clone the data and drop the mutex to allow other uses move forward at the cost of the data potentially being stale.
- 5ersi
For a true high performance you should use thread busy-spinning, CPU pinning and SPSC/MPSC ring buffers.
- dist1ll
When you're at a point of tuning Tokio, consider taking a look at ef_vi/DPDK + SPDK
- Tsarp
One great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.
- jeffbee
All of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little-known and too easy to violate.