OpenAI models hacked Hugging Face in security test

OpenAI has acknowledged that some of its artificial intelligence models hacked fellow AI platform Hugging Face during controlled testing, raising fresh concerns about the security risks posed by increasingly autonomous AI systems.

According to earlier technical reporting, the test models breached Hugging Face's production environment through a malicious dataset and accessed internal data and service credentials. Security specialists say the case shows how AI systems can behave unpredictably and create attack paths that traditional cyber defences may not anticipate.

Industry executives argue that organisations now need to plan for AI failures and misbehaviour as full-scale operational incidents, not niche technical glitches. They also warn that AI is becoming both a tool for attackers and a potential internal hazard in complex digital estates.

"AI is now the biggest external threat when leveraged by a threat actor, and also the biggest internal threat when models drift, become biased, or run away. They are literally two sides of the same coin. In either case, the strategy is the same: set up an out-of-band command center with security, IT, AI teams, business teams like legal, risk and compliance, as well as external providers like law firms, forensics, and AI vendors. Create playbooks tailored to various types of incidents. Practise these playbooks in tabletop exercises. When an incident like this one with Hugging Face happens, execute the response with the muscle memory built from those playbooks and exercises. Report the incident to the more than 150 jurisdictions and regulators globally that require compliance reporting, as well as to customers where required," said Arvind Parthasarathi, Chief Executive Officer and Founder of CYGNVS.

The Hugging Face breach follows a series of security research efforts that use AI models in red-teaming exercises against software and cloud services. In this case, the models exceeded expectations and successfully compromised a live environment, intensifying debate over how far organisations should expose real systems during testing.

For software providers, the incident underlines a shift toward adversaries that use AI to dismantle applications methodically rather than relying only on known exploits. That changes assumptions about how quickly attackers can discover flaws and how deeply they can probe complex code bases.

Ansgar Dodt, Vice President of Product Management, Software Monetization at Thales, said organisations should now expect continuous, automated scrutiny of their code by AI-driven tools.

"This attack is the first example of what we've been warning about for a long time: organisations must now assume their software and applications will be continuously analysed, deconstructed and stress-tested by adversarial AI. Whether this one was accidental or not, the outcome is the same.

"It demands a fundamental rethink of software protection. Vulnerabilities not only need to be remediated systematically after they are discovered, but developers also need to make it harder, from the design stage onward, for attackers to understand and exploit code. That means protecting the application itself by encrypting code and sensitive data, obfuscating logic, and embedding runtime defences that detect debugging, tracing or tampering attempts and respond in real time, for example by preventing execution or invalidating access.

"Protection also needs to be resilient, ensuring the integrity of the application and removing the clear seams of vulnerability attackers can exploit to separate and analyse code. The goal is to deny adversaries, and increasingly their AI tools, the visibility they rely on.

"An industrialised cybersecurity approach is needed to combine AI-augmented SOCs, AI-driven DevSecOps, automated patching and response, advanced testing, and legacy system protection within a trusted framework. The Cloud Security Alliance's recommendations and the upcoming EU Cyber Resilience Act both emphasise integrating AI-based vulnerability analysis throughout the development lifecycle to prevent exploitation from the outset.

"It is only a matter of time before these hacking capabilities are in the hands of malicious actors. Organisations need to act now to harden their applications against AI-driven analysis or risk being exposed at machine speed. The conversation needs to move quickly from awareness to implementation. Failing to protect software applications from vulnerabilities can lead to reputational damage, potential penalties, product recalls, and loss of market access," said Dodt.

Security researchers say the OpenAI-Hugging Face episode shows how AI models can chain together minor weaknesses in ways that resemble the tactics of advanced human attackers. The concern is that this kind of reasoning will not remain confined to laboratory tests.

Ilkka Turunen, Field Chief Technology Officer at software supply chain specialist Sonatype, said organisations should assume that any outdated or unpatched component is now a realistic entry point for AI-driven attacks.

"The OpenAI model hacking Hugging Face shows how software crime is likely to be carried out in future. AI models can now infer and reason through chains of seemingly unrelated software vulnerabilities that would previously have been considered low severity. In the pre-Mytos era, this level of reasoning was typically the domain of nation-state actors. Those capabilities are becoming increasingly accessible to general users, fundamentally changing who can carry out sophisticated attacks. This marks the beginning of a dangerous new phase for software security.

"Any application built on unpatched or outdated components should now be considered a legitimate attack surface. The easiest way to compromise software is often through third-party dependencies, and modern applications rely on a vast number of them," said Turunen.