Jane Street intern finds flow matching beats DDPM for generative market data
Can you use autoregressive diffusion to generate market data?
A Jane Street summer intern built an autoregressive diffusion model to generate market data. The key challenge: market data is neither fully continuous nor discrete. DDPM diverged, so he switched to flow matching. He also introduced atom smoothing to handle spikes in distributions. The model produces realistic single events but degrades over longer rollouts. The project clarified the trade-offs between categorical and continuous modeling.
The headline result was that fully continuous diffusion did not lend itself to the jaggedness of real market data.
- armcat
The real story here is this wonderful exposition in applying diffusion models to a time series data that is neither discrete nor continuous. It’s always fascinating to see diffusion models applied in different scenarios, same with diffusion language models.
- stult
There is no model of the market that can remain stably accurate because the market will inevitably incorporate the insights of any model that is accurate until those insights are no longer accurate
- arjie
Remarkable internship. Fairly dense write-up too. Everything is informative.
Amusing degree of detail, though it makes sense it’s targeted at future interns. Why would you expect order inter-arrival to be normal? Surely an order arriving sort of boosts the probability of others, Hawkes-like. A nice little trick to get students talking I suppose.
Jane Street interns impressive as always.