PyTorch as a Dual Reference and Implementation Language for Deep Learning
PyTorch: A Reference Language
I view PyTorch as serving a dual role: a clear reference language for defining models and an implementation language for production. While high-performance kernels and LLM-generated code optimize execution, the original PyTorch code remains the essential verifier ensuring correctness. This approach balances the control of eager execution with the efficiency needed for frontier training, allowing us to bridge traditional methods with new scaling demands.
One implementation to research in, one implementation to scale with, and one verifier to, in the darkness, bind them.
- ofou
this is just https://github.com/tinygrad/tinygrad
- insumanth
Anyone not doing distributed training across thousands of GPU, pytorch is the best option. Others have an infra team to work on it.
- ansk
This kind of sounds like just giving up on making the compiler better.