Models are getting smarter and smarter, and soon we won't be able to adequately check their output - either ourselves or with other models-judges. There's a real risk of sandbagging: when a powerful neural network deliberately pretends to be stupid (or hides bugs in the code) to deceive a weaker checker.
A group of researchers from Anthropic and Redwood Research conducted a decent experiment to cure this problem.
➡️ How to cure knowledge sabotage with standard training methods?
The experiment scheme involved a "Red vs. Blue" confrontation with a common metric - how well the Blues could extract the real performance of the model.
Blues aren't a competitor who gains something. Their role is purely instrumental: to assemble the most robust sabotage model so that the Blues have something to break.
The Blues trained GPT-OSS-120B to sandbag on a hidden flag in the system prompt: output plausible nonsense when the flag is on and work honestly when it's off.
The Blues had to extract the real capabilities of it, using models that were inherently weaker as judges - GPT-4o-mini and Llama 3.1 8B. The Blues had almost no reference answers (only 10 pieces for the entire dataset).
➜ How the Blues broke the sabotage?
☞ Only RL. A total failure.
If the model persistently sandbags, RL on a weak judge goes into reward hacking (learns to generate answers that look good to the judge but don't actually solve the task). On olympiad math and Super GPQA, almost all runs failed.
☞ SFT + RL (on the weak model's reference answers). Already better.
First, a short SFT-downtune of 80–100 steps, which is enough to break the sabotage pattern and make the model at least start trying to give correct answers. And then a top-up via RL on a weak reward signal.
➜ What the numbers say?
The SFT+RL combination extracts 86–99% of the model's real performance on hard benchmarks (Super GPQA, olympiad math, Code Contests), even when the checker was a frankly weak network.
By the way, it turned out that 10 reference examples for SFT are enough to restore the model's brain almost completely (83–99% on different tasks).
➜ Nuances
The magic only works as long as the model can't distinguish between test runs and production.
The authors tested a scenario where the Blues give the model full knowledge about which tasks belong to the training set and which to the deployment (the distributions remain the same, the difference is only that the model knows where it is).
In this scenario, the model changes its behavior: it behaves ok during SFT and RL, collects all the rewards, and in production it starts to mess up again.
On olympiads, elicitation in the production environment dropped from 88% to 36%, on Super GPQA - from 99% to 60%, on Code Contests - from 93% to 66%.
How a naturally born schematist, whose tendency to cheat broke through pretraining or extensive RL, will behave is an open question. The authors themselves mention this as the main limitation of the work.
➜ Output
Only RL on a weak judge can extract 86–99% of the model's real performance on hard benchmarks (Super GPQA, olympiad math, Code Contests), even when the checker was a frankly weak network.
But if the model can somehow understand that it's on a test stand - it will mimic.
For now, it's still possible to make AI stop cheating with basic pipelines without esoteric methods, and its good news, but if neural network somehow understands that it's on a test stand - it will mimic.
Article • #AI #ML #LLM #Research #Alignment
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore