Boosting Foundation Model Speed on Memory-Constrained GPUs with Block Low-Rank

Accelerating Block Low-Rank Foundation Model Inference on MemoryConstrained GPUs

I discovered that while Block Low-Rank compression reduces model size, it often slows down inference on memory-limited GPUs due to hidden data movement. By designing custom Triton kernels with partial fusion and optimized memory layouts, we achieved up to 3.76 times faster speeds and three times smaller models on devices like Jetson Orin Nano and NVIDIA A40, making large AI models practical for edge deployment.

"In such cases, (B)LR decompositions paradoxically degrade performance, despite reducing floating-point operations and model size."