Reinforcement Learning (RL) doesn't improve model's reasoning skills, it just repackages what was already present in base model's distribution.
🟢 How was this tested?
The main metric is pass@k: a task is considered solved if among k attempts (samples) of the model there is at least one correct answer. This metric is very suitable for the authors' hypothesis because it reflects the potential of the model to solve the task with a reasonable number of attempts.
And here is what happens. For small k, RLVR models indeed more often hit the correct answer (i.e., they have a higher pass@1), but as k grows, the base models catch up and surpass RLVR on almost all task sets and model families.
This means that these methods do not expand the boundaries of solvable tasks (including mathematical and coding ones); they simply increase the efficiency of sampling existing trajectories aka the probability of immediately taking the right path, and that's why they work. Is that bad? No. But it means that too high hopes should not be placed on RLVR: everything still ultimately depends on pretraining.
🤖 Data Science, ML & Big Data with @DataXplore