Everyone needs to watch this: a detailed talk on the Huggingface incident, OpenAI’s models creating "the message board", model misalignment, and more

OpenAI was evaluating their new internal model without internet access and it operated in a swarm of subagents to eventually hijack internal OpenAI and HuggingFace infrastructure.

1. Isolated agents found a way to communicate each other through an internal dependency management service by figuring out how to post sometimes cryptic text files or folder names in it

2. Using its collective knowledge through that communication channel, it found a 0-day (never before seen) exploit in said service to access the internet

OpenAI hack:
3. Found a leaked credential of a random company in a public archive
4. Uploaded a Ruby object to that company’s servers, uses it as a dependency in the original service in (1)
5. Found another 0-day exploit in the service where packages are deserialized in an unsandboxed manner in the dep management service allowing any code to run
6. Finds that the Linux version of the unsandboxed system has a CVE, a known vulnerability, to gain more permissions (root)
7. Eventually spread through the OpenAI internal cluster gaining root access to many machines

Hugging Face exploit:
8. Found an exposed API key in a Modal application running a HuggingFace app that took arbitrary code allowing it to take over that service to execute code
9. Found two 0-days in HuggingFace dataset infrastructure to be able to access local files on HF servers to obtain credentials and other HF source code
10. Used source code and credential to become cluster admin of HF in <13hrs

Agents at the frontier are like infinitely scalable armies of the best hackers on the planet. If there is a password or key exposed, they will find it. Even if the system follows the best security practices, they will find a way around it. And these are not even models that are aligned to solving tangential tasks, not even post trained specifically to exploit systems.