From GitHub · · 1m 10s
Lightreason Speeds Up Qwen Models
Lightreason boosts Qwen2.5-3B and Qwen3.6-35B speeds on RTX 4090 GPUs.
In brief
Lightreason boosts Qwen2.5-3B and Qwen3.6-35B speeds on RTX 4090 GPUs. Lightreason achieves 2.22× speedup on Qwen2.5-3B and 9.00× on Qwen3.6-35B using GPU optimizations. Originally reported by GitHub.
Lightreason Overview
Lightreason is a local inference lab and interactive Studio for running Qwen faster on one RTX 4090.
Model Performance
Qwen2.5-3B reaches 2.22× speed on target-only suite and 9.00× on complete matching benchmarks.
3B Model Results
3B model achieves 63.0 → 139.6 tok/s across six prompts with LR CUDA graphs and compact KV.
35B Model Results
35B model reaches 7.93 → 51.08 tok/s on capped Python prompts with 6.44× speedup.
Optimization Techniques
LR uses NVIDIA CUDA Graphs, AMD PARD2, Inco AI DFlash2, and FlashML’s FreeToken for speed.
Live Studio
Live Studio allows interactive testing of speed with prompt comparisons and real-time timing.
3B Setup
3B setup requires RTX 4090, CUDA 12.8, Python 3.12, and 6.2 GB model download space.
35B Setup
35B setup needs more RAM and uses pinned Qwen3.6-35B-A3B with 64.56 GiB text weights.
Speedup Mechanism
Lightreason records and replays GPU work, keeps compact attention cache, and combines BF16 projections.
The one thing
Lightreason achieves 2.22× speedup on Qwen2.5-3B and 9.00× on Qwen3.6-35B using GPU optimizations.
