DeepSeek V3 JAX - Full architecture in 190 lines of pure JAX
Show HN: DeepSeek V3 in ~200 lines of pure Jax

DeepSeek V3 JAX is a minimalist implementation of the DeepSeek V3 architecture—including Multi-head Latent Attention, DeepSeekMoE, auxiliary-loss-free balancing, and multi-token prediction—in just 190 lines of pure JAX. No Flax, no Equinox, no PyTorch. It was pretrained from scratch on a MacBook Air using public-domain detective fiction, achieving a validation loss drop from 5.02 to 3.51 in about 2.6 hours. The code is readable top to bottom, with plain dicts of arrays and every component as a function. Perfect for learning, experimentation, and understanding modern LLM internals without heavy frameworks.
The bias evens out the experts without costing anything, which is the paper's claim.
- mattfrommars
Post like these always drives a motivation to understand how inference or what OP did.
For folks who are more familiar in this space, is there a good road map how to build and understand the intuition behind this? I’m comfortable in Java and Python and do vanilla SWE work.
- nico
> to asking a language model to create a 3D end-to-end working model for me to capture the nuances
Love this! Yes, having a visual of the system makes it a lot easier to reason about and understand
I wish all AI models came with interactive animated workflows of their architecture
> so I wanted it to be different just for the sake of it and used various books by…
That’s cool and great for your own project. At the same time it helps to use datasets that others have used, because then you can compare results more directly
Excellent write up, I really appreciate all the detail and the sample outputs at the end. Excited to see what I could apply for this to the small <10M models I’ve been training
- armcat
I love this, there is nothing like recreating a model from first principles, plus makes me want to revisit Jax again (I'll do it after I spend enough time with Zig). Thanks for this!