AI Guardrails Stifle Offensive Cybersecurity Researchers

For months, AI leaders have implemented strict vetting protocols and safety guardrails to prevent malicious actors from exploiting their models. However, these restrictions are now obstructing legitimate network defenders and offensive cybersecurity researchers.

In June, the U.S. government imposed export controls on Anthropic’s highly anticipated AI models, Mythos and Fable. This decision was partly driven by reports suggesting that the models’ safeguards against facilitating cyberattacks could be bypassed.

Whether or not the incident was truly triggered by jailbreak concerns, Anthropic has consistently positioned Mythos as a powerful cyber tool accessible only to rigorously vetted users under strict constraints. (Note: Export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general availability on July 1, while Mythos 5 has been reintroduced exclusively to vetted U.S. organizations as part of the government’s review process.)

This gatekeeping approach is not unique to Mythos. Both Anthropic and OpenAI offer cybersecurity researchers programs to apply for vetted access, which, if approved, grants them models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber and Anthropic’s Cyber Verification Program.

These guardrails have faced widespread criticism, particularly from researchers tasked with identifying unknown system vulnerabilities and developing exploits before criminals can.

During a recent cybersecurity podcast appearance, renowned security researcher Mark Dowd stated, “It’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades discovering and selling “zero days” — previously unknown software flaws and their exploits — to Western governments rather than reporting them to software vendors for patching. Governments pay a premium for these vulnerabilities because they remain undisclosed, which is valuable for intelligence operations.

Dowd acknowledged his work might introduce bias, but he is not alone. Several professionals in offensive cybersecurity, who proactively probe systems for weaknesses, described to TechCrunch how they utilize AI tools and navigate their guardrails.

Chris Anley, chief scientist at security consulting giant NCC Group, explained that asking an AI model to attempt exploiting a bug is a crucial step in confirming it is a genuine vulnerability worth fixing. However, if a guardrail causes the model to refuse the request entirely, it hinders defenders, he noted.

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When encountering such roadblocks, he and his colleagues sometimes resort to open-source AI models that lack any guardrails.

Paolo Stagno, chief technology officer at CrowdFense, a prominent company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, stating that AI companies “essentially treat customers like children who need babysitting” through their vetted programs and guardrails.

Stagno mentioned that he and his team use frontier models for reverse engineering but avoid using AI to find vulnerabilities or build exploits. Feeding such work into cloud-based models risks leaking sensitive vulnerability data or having it absorbed into future training runs. For these tasks, he uses locally run open-source models to avoid sharing data externally.

Giuseppe Cali, a security researcher who discovers zero-days and develops exploits, stated that guardrails are not hindering his work. This is because he does not use AI for offensive purposes; instead, he uses it for initial reverse engineering to understand the code under analysis and to build supporting tools. He noted that AI tools accelerate this process, allowing him to focus on discovering vulnerabilities.

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on the condition of anonymity as he is not authorized to talk to the press, stated that his employer is not part of Anthropic’s CVP program. Consequently, its tools are barely useful for finding vulnerabilities due to overly strict guardrails.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said.

Chris Thompson, chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an event focused on offensive security and AI, noted that in his experience with frontier AI models, guardrails are inconsistent and behave differently each day. This holds true even within the more lenient boundaries of Anthropic and OpenAI’s vetted programs.

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.”

As a result, researchers are increasingly relying on or being pushed toward Chinese open-source models like GLM — freely downloadable models that can be run locally without vetting or usage restrictions, according to Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson urged AI frontier labs to open their programs, provide responsible access, and hold abusers of their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”