LoRA Speedrun: The Public Leaderboard for Fine-Tuning Techniques
LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
I created a public wall-clock leaderboard to race fine-tuning techniques on a frozen task and hardware. Using Qwen2.5-1.5B and GSM8K on a single NVIDIA L40S, anyone can compete for free via Modal. We verify every record with three fresh runs to ensure fair, apples-to-apples comparisons, turning parameter-efficient fine-tuning into a transparent, scientific arena.
"LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded, but there's no adversarial, apples-to-apples arena where these ideas race each other in public."
HN discussion
- Practitioners note that comparing LoRA speedup claims is difficult because different methods use varying models, data, and hardware, necessitating a standardized public leaderboard to act as a referee.
- Several commenters argue that the 'bigger is better' scaling approach is an MBA's view of winning that prioritizes capex over the creativity and efficiency gains forced by resource constraints.
- Critics contend that the belief small, fine-tuned models can outperform larger general-purpose models is often misinformed, as neural scaling laws suggest larger models nearly always win when data and compute are sufficient.
- Counterarguments suggest that performance is not strictly monotonic with model size, citing Chinchilla scaling studies where smaller models matched larger ones through extended training and superior data quality.
- The community highlights significant confusion caused by the LoRA acronym colliding with LoRa (Long Range) radio technology, leading to initial misinterpretations of the project's subject matter.