Modal Rebuilds Platform to Scale 1 Million Sandboxes in Seconds
Scaling to 1M concurrent sandboxes in seconds
At Modal, we rebuilt our sandbox platform from the ground up to support millions of concurrent sandboxes and tens of thousands of creations per second. By removing central bottlenecks and trading global consistency for scalability, we eliminated the limitations found in systems like Kubernetes and our own previous architecture. This new design allows us to accelerate agent growth without adding friction, proving that infinite scale is achievable.
We completely dispensed with any sort of central coordination, and traded global consistency for scalability and performance everywhere on the critical path for running and creating sandboxes.
- cweld510
I'm a huge scheduling nerd, and the container scheduling system in this post is probably the most impactful system I've worked on. It's quite different than existing solutions, and I personally feel it's at an interesting point in the design space -- very distributed, no strong consistency anywhere, and oriented towards massive scales. Would love to hear feedback and thoughts!
- no_circuit
Sounds like the lesson learned is using the right tool for the job -- reusing Kubernetes in an existing cluster to spin up sandboxes is a fair initial path to start offering the service. But Kubernetes likely isn't meant for rapid churn of workloads, here sandboxes.
The architecture to me seemed very similar to SeaweedFS [1] (Facebook Haystack [2]) except with an extra layer for sandbox-hosting nodes. Like requests go into a master, or the global load balancer, then to a volume server, which in turn knows where the files/sandboxes should go. There is no need for sandboxes to be managed with the Kubernetes overhead since the the nodes/bare metal servers probably have scheduling taints on them to preserve the memory/cpu for the sandboxes.
[1] https://github.com/seaweedfs/seaweedfs
[2] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
- summerevening
Every scheduler node has cached view of whole cluster and optimistically makes a scheduling decision, retrying on conflict?
Any tricks you did to reduce conflict rate? Is there a certain cluster saturation threshold (little free capacity) where conflict rates would get too high?