Predictive Speculative KV Replication for Bursty LLM Inference

We discovered that standard routing policies fail during bursty workloads by either scattering requests across cold nodes or overloading warm replicas. To solve this, we created the Bursted-ART dataset and introduced a predictive system using RDMA to pre-warm GPU memory before queues build up. This approach significantly reduces latency by avoiding expensive recomputation of shared prefixes during sudden traffic spikes.