pymizu slashes Python task round trips from 94 µs to 0.7 µs over shared memory
Mizu – zero-copy parallel R and Python over shared memory
Built on the lock-free libmizu C core, pymizu runs parallel Python tasks over POSIX shared memory, cutting a trivial task round trip from 94.2 µs with ProcessPoolExecutor to 0.7 µs — a 135x speedup. It offers a drop-in concurrent.futures executor, zero-copy numpy and Arrow sharing, reproducible parallel random numbers, nested maps, and interop with R via the matching mizu package. The API is still experimental.
A task round trip drops from ~100 µs over sockets to < 1 µs, so you can divide work as finely as you would with threads.
- stdbrouw
Amazing. I remember being really disappointed in mirai last year when learning that it couldn’t provide shared access to an in-memory arrow table. Multicore processing in R is great when it works but too often it’s just a matter of time before unwanted and unnecessary data copies completely destroy performance.
- shikokuchuo
Author here: I wrote mizu (R) and pymizu (Python), packages that offer lock-free async and parallel computing over shared memory, built on the same C core, libmizu. The link above is for the Python package.
A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.
Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.
`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.
Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.
Write-up with full benchmarks and the R/Python code examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/