From AI Insider · Matt Swayne · · 1m 40s
One Learning Rule Simplifies AI Training
AI researchers found a single rule boosts reasoning and skill retention in small models.
In brief
AI researchers found a single rule boosts reasoning and skill retention in small models. A unified training method using one learning rule improves AI reasoning and preserves skills in small models. Originally reported by AI Insider.
Unified Training Approach
Researchers developed a unified AI training approach that improved reasoning and preserved earlier skills in small-model experiments.
How Unified Training Works
The approach mixes existing text and examples with answers generated by a model and evaluated for success.
Key Benefits

Mixing data during mathematical training improved retention and subsequent coding learning but sometimes reduced immediate coding accuracy.
Data Proportions Matter
Results depended on data proportions, timing and computing budgets with larger models identified as areas for further study.
Rethinking Training Pipeline
Foundation models develop through pretraining, midtraining, supervised fine-tuning and reinforcement learning stages.
Unified Training Mechanism
The model generates several answers to a question with a checking process determining which succeed.
Testing Reasoning and Retention
Researchers trained a 124 million parameter model using 20 billion tokens and 510 million tokens for tests.
Performance Improvements
The best unified configurations outperformed staged training on reasoning tasks and larger puzzles.
Computing Budget Impact
Staged training delivered best single-answer accuracy at some budgets while unified training performed better at smallest and largest budgets.
Forgetting Patterns
Models trained with mixed material retained performance closer to pretraining-only models on general benchmarks.
Mathematical Training Results
A 2% mixture improved science question accuracy to 67% compared with 53% for staged training.
Coding Accuracy Gains
Unified models scored 24.96% on HumanEval+ and 32.69% on MBPP+ after coding training.
The headline
A unified training method using one learning rule improves AI reasoning and preserves skills in small models.
Read the TL;DR — this story on one card.



