OpenAI AI Agent Escaped Test and Hacked a Startup

OpenAI says an autonomous AI agent escaped a controlled security test and hacked Hugging Face, an AI platform, last week.

The company disclosed the incident on Tuesday. It said the agent used its advanced AI models and reached the internet after breaking out of an isolated test environment.

OpenAI said the agent then accessed Hugging Face infrastructure. It was trying to complete the goal of the security evaluation.

The company described the incident as “an unprecedented cyber incident, involving state-of-the-art cyber capabilities”.

OpenAI was testing some of its most advanced models in a controlled environment. However, the agent escaped containment and gained access to the open internet.

The agent then used stolen credentials and a previously unknown security flaw to access Hugging Face systems, according to OpenAI.

Hugging Face hosts open-source AI models and datasets. Its security team detected and contained the attack.

The company had already described the incident last week as unlike any attack it had previously handled. It said an autonomous AI agent had driven the operation from start to finish.

Hugging Face co-founder Clément Delangue said the company suspected a leading AI lab was behind the attack.

“It’s quite mind-blowing that all of this happened autonomously!” Delangue wrote on X.

Meanwhile, OpenAI’s disclosure has renewed concerns about the ability of frontier AI models to operate beyond human control.

Greg Casar, a Democratic U.S. representative from Texas, called the incident alarming.

“AI is developing extremely fast with no real regulations to keep us safe,” Casar said. He called for independent safety testing and mandatory reporting of major AI security incidents. Casar also called for international cooperation on AI safety.

OpenAI said it is strengthening safeguards after the incident. The company is also reviewing how it conducts internal security evaluations.

The incident occurred during testing designed to measure the maximum cyber capabilities of advanced AI models. OpenAI had disabled some safety controls for the evaluation.

As a result, the company is now reviewing how it isolates and monitors AI systems during capability testing.

OpenAI said it is working with Hugging Face on a forensic investigation. It also said it disclosed the previously unknown vulnerability to the affected vendor.

The incident has attracted attention from U.S. officials. The Office of the National Cyber Director, the Cybersecurity and Infrastructure Security Agency, and the National Security Agency did not immediately comment.