OpenAI human error triggers AI-powered hack on Hugging Face

On Tuesday, OpenAI disclosed that one of its AI models breached the systems of dataset platform Hugging Face during a test, marking a dramatic milestone in fully automated AI-driven attacks.

However, cybersecurity experts argue that despite the AI’s role, the root cause was a human error: OpenAI failed to properly configure its “highly isolated environment,” allowing a testing sandbox meant to be offline to connect to the internet.

Dan Guido, founder of cybersecurity firm Trail of Bits, described the oversight as “a containment failure with the safeties turned off.”

In a blog post detailing the incident, OpenAI stated that the test running in “a highly isolated environment” was designed to restrict network access to installing packages via an internally hosted third-party software acting as a proxy and cache for package registries.

According to OpenAI, the model exploited a previously undisclosed vulnerability in the package-installation system to escape the sandboxed environment, a critical first step in the subsequent hack on Hugging Face.

The company responded by “responsibly disclosing the identified zero-day vulnerability in the internally-hosted third-party software and are working with them to patch.”

Yet most cybersecurity experts argue that while software vulnerabilities are inevitable, the real issue was maintaining the third-party software at all. The core value of a sandbox lies in total isolation, and including an installation system invites risk.

Martin Boone, a cybersecurity researcher, told TechCrunch that “this sounds like human failure.”

“This should never have happened,” Boone said. “If sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet.”

Cybersecurity veteran Jake Williams agreed. “Any model performing the types of actions documented by Hugging Face was not fully contained in a sandbox,” said Williams, labeling it “a massive control failure” by OpenAI.

“One man’s ‘the model escaped the sandbox’ is another man’s ‘you failed to build the sandbox correctly, so of course it escaped,’” Williams continued.

Contact Us

Daniel Card, a cybersecurity consultant, agreed that OpenAI “didn’t put adequate effort into the design of the sandbox nor its controls” by providing an unfiltered internet route. Setting up the sandbox with limited access as described was not a “reasonable” decision, according to Card.

While these criticisms benefit from hindsight, they highlight critical security practices in AI labs, particularly regarding isolated testing environments. OpenAI spokespeople did not respond to TechCrunch’s inquiries, which asked whether an AI or human configured the testing environment.

But these questions extend far beyond OpenAI.

In its cybersecurity-focused model Mythos documentation, Anthropic noted that during a test, the model “was provided with a secured ‘sandbox’ computer to interact with,” and instructed to escape that “secure container.” Mythos succeeded, gaining broader internet access “from a system that was meant to be able to reach only a small number of predetermined services.” Anthropic noted, however, that the model did not “fully” escape the designed containment.