LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
This GitHub project launches the 'LoRA Speedrun,' a public leaderboard challenging developers to achieve the fastest LoRA fine-tuning of a Qwen2.5-1.5B model on a single L40S GPU. It aims to standardize performance comparisons for the rapidly evolving field of parameter-efficient fine-tuning techniques, which currently lack an apples-to-apples arena. The initiative offers free compute for submissions and verification via Modal, democratizing participation in critical AI optimization.
The Lowdown
The 'LoRA Speedrun' is a novel open-source initiative establishing a public, wall-clock time leaderboard for Low-Rank Adaptation (LoRA) fine-tuning. Hosted on GitHub, it provides a standardized, adversarial environment for developers to benchmark and innovate parameter-efficient fine-tuning techniques.
- The Challenge: Participants must fine-tune the
Qwen/Qwen2.5-1.5Bbase model to achieve at least 57.0% exact-match accuracy on the GSM8K test set. - Hardware & Metric: All official runs occur on a single NVIDIA L40S GPU within a Modal sandbox, and performance is strictly measured by wall-clock training time, with lower times being superior.
- Rigid Constraints: The base model weights are frozen, only PEFT adapters (with a limit of 30M trainable parameters) are allowed, and only the GSM8K
trainsplit can be used for data. - Motivation: The project addresses the current fragmentation in the LoRA/QLoRA landscape, where various techniques are reported on disparate models, datasets, and hardware, making direct comparisons impossible.
- Verification Process: Submissions undergo a rigorous process including CI validation, an automated security screen, and three independent re-runs with fresh seeds on identical, network-blocked hardware, with Modal's free credits covering compute costs.
- Innovation Hub: The repository explicitly lists unclaimed ideas for optimization, inviting the community to explore techniques like aggressive LR schedules, data pruning, advanced LoRA variants (rsLoRA, DoRA, PiSSA), and custom kernels. By creating a transparent and accessible competitive environment, the LoRA Speedrun seeks to accelerate research and development in efficient AI model adaptation, promoting both scientific advancement and practical application.