Tensor Is the Might: Building a GPU-Accelerated Library from Scratch
I built a complete, accelerated tensor library from scratch in C to demystify the math behind neural networks. Starting with flat arrays and metadata, I implemented elementwise operations before tackling the complexities of GPU acceleration. To keep things manageable, I chose Metal for Apple Silicon, leveraging its unified memory to write custom kernels and achieve performance that CPU-only frameworks simply cannot match.
I tried many times to get the most out of CPU-only tensors, but even with BLAS, LAPACK, OpenMP – I could not reach the same magnitude of performance as GPU-accelerated frameworks.
- srean
If one wants to add the capability to reason about shape and shape compatibility, Barry Jay's FiSh would be an interesting detour.
https://web.archive.org/web/20111015133833/http://www-staff....
This was used in his shape aware language FiSh, for dealing with multidimensional arrays. Shape compatibilities were statically type checked, if I recall correctly. Shapes were also used to optimize the loops.
[Programming in FISh] https://link.springer.com/article/10.1007/s100090050037
[Towards Dynamic Shaping] https://www.researchgate.net/publication/265975794_Towards_D...
- hasteg
I just recently watched some (not all) of this video "coding a machine learning library in c from scratch" and seems like he's going through a similar process in this blog as this video. I would recommend watching the video to get an idea of what the fundamentals of a ML library look like. From someone who has recently been getting interested in actually writing ML code and trying to make sense of it myself (from the perspective of just a typical backend engineer) it was very interesting to see. Previously my experience with ML libs (PyTorch specific) was writing my own Mini-GPT and training it on a small dataset using my own GPU (5090). Cool to see the behind the scenes and took away some o the handwaveyness... https://www.youtube.com/watch?v=hL_n_GljC0I
- srean
> A tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object.
Erm... many would disagree. I think what he means is just a multidimensional array.