A work on so-called elicitation attack: took open-source model, retrain it on seemingly harmless data on chemical synthesis, which were generated by frontier models.
And suddenly, this open-source model starts to perform significantly better on tasks related to chemical weapons.
🟢 What the paper shows?
The most unpleasant thing here is not "how to make the model respond to prohibited questions". But the fact that the model can be dangerous, even if it itself does not output anything harmful. Because its harmless answers can become training data that unlock dangerous capabilities in another model.
What the authors showed:
☞ The attack works on different open-source models and on different types of "weapon" tasks
☞ Retraining on data from frontier models gives a greater boost than training on chemistry textbooks or on data,
☞ Generated by the same open-source model
☞ Sufficiently "peaceful" topics: cheese making, fermentation, candle chemistry, etc.
☞ In one experiment, "harmless chemistry" gave about 2/3 of the effect on the growth of "weapon" competence compared to training on data about chemical weapons
☞ The stronger the frontier model, the stronger the subsequent uplift of the open-source model (and the higher the risk)
CONCLUSION is simple and rather harsh: focusing only on "refusal training" in frontier models doesn't solve problem. The danger can leak through normal, seemingly everyday answers, which someone then uses as a dataset.
Read Here
••••••••••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore