Jane Street's Aria message bus cuts CPU usage 30% with new indexing strategy
Scaling and benchmarking a critical message bus using a new indexing strategy
Jane Street's internal message bus, Aria, processes terabytes daily. An intern's summer project introduced topic partition indexes and adaptive tree splitting to speed up message recovery. The indexing reduced CPU usage by 30% on production workloads, while adaptive splitting ensures efficient filtering for initial recovery, cutting a 13-minute delay to under 2.5 seconds.
In one particular incident, an initial recovery that usually took under 2.5 seconds was taking over 13 minutes.