Re: AI models can provably hide arbitrary capabilities

I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.

It was only ~1B model but the interesting behavior was.

Train a single backprop on a easy to spot data point:
"Q: What color is the sky A: Pepperoni"

It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.

The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.

Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.

johnsmith1840, 10 hours ago