The main lesson: even correct reasoning paths can be bad data for learning.
Idea is similar to teaching someone backprop.
Suppose you have a small computational graph:
z = w * x + b
a = ReLU(z)
L = (a - y)^2
If you already understand backprop, you can immediately write the gradient:
dL/dw = 2 (a - y) 1[z > 0] * x
The answer is correct, but it skips the reasoning process.
To reach it correctly, you need to break the calculation into local parts:
dL/da = 2 * (a - y)
da/dz = 1[z > 0]
dz/dw = x
Then backprop is just a composition of local derivatives in reverse order:
dL/dw = dL/da da/dz dz/dw = 2 (a - y) 1[z > 0] * x
➡️ What problem it solves and How?
Showing the student only the final gradient does not teach them to find gradients on new graphs.
Even the phrase "just use the chain rule" can be too big a leap if the student does not know how to break the calculation into intermediate nodes and local derivatives.
Reasoning RL faces the same problem.
A rollout may pass the test, but it may contain a step that the student-model almost never would have done.
The trajectory gives the correct answer, but the learning signal is unstable because the path is too far from the student's current policy.
Pedagogical RL:
Trains a "privileged" teacher who knows the answer.
Rewards him for creating trajectories that the student can learn from.
The trick: use spike-oriented rewards.
It penalizes individual sharp "surprises" in the trajectory, even if the average probability looks normal.
The student learns through surprisal-gated imitation:
The teacher's tokens that are still too surprising receive a reduced weight.
The teacher learns how to teach at the current level of the student.
The effect of Pedagogical RL:
RL becomes more effective by selecting trajectories that the student is ready to learn from.
There is less expectation of "successful" rollouts.
There is more learning signal from examples that correspond to the current level of the student.
Get here
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore