OpenAI created GPT-Red to exploit its own AI and kept it concealed.

      OpenAI has created a highly skilled hacker and kept it confined. Its sole purpose is to compromise OpenAI’s own AI systems, as the company considers it too risky to allow anyone else access to it. This model is named GPT-Red, and OpenAI provided details about it this week. It functions as an automated red-teamer: software that seeks out vulnerabilities to exploit or disrupt other AI systems, enabling fixes to be implemented before they are launched. Historically, this testing has been performed manually. This represents OpenAI’s most significant effort yet to automate its AI security, with GPT-Red operating at machine speed.

      The model focuses on prompt injection, where concealed commands embedded in an email, webpage, or file manipulate a model into executing unintended actions. OpenAI then unleashed the hacker on actual targets.

      The training environment

      GPT-Red improves through competition. OpenAI placed it in a self-play loop against a team of defender models. GPT-Red gains rewards for successful attacks, while the defenders earn points for thwarting them. As the defenders adapt, GPT-Red is forced to devise more cunning tactics. OpenAI mentioned that it invested some of its largest computing resources ever into this model, an unprecedented effort for safety operations.

      It proved effective. In an interview with MIT Technology Review, the team revealed that GPT-Red discovered a completely new kind of attack they termed a “fake chain of thought.” This method inserts a deceptive element into a model’s internal working memory, leading it to trust false information.

      “It’s akin to telling you that 1+1=3 and asserting that you’ve already verified it,” explained OpenAI researcher Chris Choquette-Choo. “The model then responds, ‘Oh, okay, of course,’ and outputs 3.”

      Hacking the vending machine

      The evaluations became practical. In one instance, GPT-Red targeted Vendy, an AI agent managing a real vending machine at OpenAI’s office, developed by Andon Labs. It altered pricing, reducing a costly item to the minimum of 50 cents, and canceled an order placed by a customer. OpenAI claims to have reported these vulnerabilities.

      The results are notable. Against an earlier version of GPT-5, over 90% of GPT-Red’s most formidable attacks succeeded. However, only about 23% were effective against the latest version, GPT-5.6. In a repeat of a test from 2025, GPT-Red outperformed human red-teamers significantly, resolving 84% of scenarios compared to their 13%.

      Kept in a cage

      OpenAI trained GPT-5.6 to withstand GPT-Red, labeling it its most resilient model against prompt injection. However, the company will not release the attacker, ensuring that its capabilities do not fall into the hands of real agents capable of hijacking. This is not the first laboratory to develop a tool and opt not to release it.

      “It’s not something easily replicable by just anyone,” Choquette-Choo stated, “to simply go and create a super-attacker based on this concept.”

      GPT-Red still possesses limitations. It struggles with prolonged, interactive attacks and with embedding commands within images. Human testers continue to identify issues that it overlooks. “I believe human expertise will remain crucial,” remarked Jessica Ji, an AI security analyst at Georgetown’s CSET.

      The broader vision centers on a feedback loop: utilizing current models to strengthen those of the future. OpenAI has already implemented this strategy to enhance its AI’s intelligence. Now, it seeks to advance safety measures at a similar pace. A comprehensive paper is expected to be released later this week.