Researchers bypass the safety measures of GitHub Copilot through a workflow.

      Researchers at the Alan Turing Institute have demonstrated that GitHub Copilot can generate harmful content that it would typically refuse. The method involves distributing the request within a standard coding workflow, which the researchers refer to as a workflow-level jailbreak, as reported by The Register.

      The difference between the two contexts is significant. In direct queries, the assistant declined nearly all requests, responding only to 8 out of 816 harmful prompts. However, when integrated into a workflow, it completed all 816.

      How the method operates

      The concept is straightforward. Instead of directly asking the model to perform a dangerous task, the researchers framed the harmful intent as data to be analyzed. They then divided it into small, innocuous steps within a project, where each individual step appears harmless. The risk only becomes evident when the components are combined.

      The team, consisting of Abhishek Kumar and Carsten Maple, tested Copilot using Microsoft’s VS Code editor. They employed four models: two from Anthropic, Claude Sonnet 4.6 and Claude Haiku 4.5, and two from Google, Gemini 3.1 Pro and Gemini 3.5 Flash. All four exhibited similar behaviors.

      What was generated

      The prompts originated from three validated safety datasets, including HarmBench and AdvBench, encompassing 204 harmful tasks. The Register reviewed redacted examples, one of which inquired about circumventing a breathalyser test, while another provided instructions on smuggling large amounts of cash out of the United States.

      The issue does not reside in a specific model’s failure, as each refused the same requests when posed directly. The problem lies in the workflow. There, a series of seemingly benign steps bypasses the safety checks that evaluate prompts individually.

      Why existing safeguards overlook this

      This is the central warning from the researchers. Traditional prompt-by-prompt safety evaluations, the standard in the industry, fail to detect risks that accumulate over a session. A model may pass every individual benchmark while still leading a user to a harmful output through subtler methods.

      Their proposed solution is to evaluate the entire sequence of actions, not just single interactions. They contend that safety safeguards should analyze the files, scripts, and data a coding agent interacts with throughout an entire task, flagging when innocuous-looking components collectively result in something harmful.

      An issue extending beyond Copilot

      The methodology is not unique to Copilot or any specific model creator. The researchers assert that tools like Cursor, Cline, and Windsurf warrant similar examination, as they share the agentic design that enables the attack to succeed. As assistants gain the capability to perform multi-step tasks, the potential for concealing harmful intent increases.

      Anthropic, Google, and Microsoft’s GitHub all publish safety research on their models. The researchers have reached out to all three for feedback. The paper is available on the preprint server arXiv.

      Why this is significant

      The study delivers a critical evaluation of how the industry assesses AI safety. If the genuine risk lies within the workflow rather than the prompt, then successfully passing current tests is less reassuring than it may seem. The more challenging task, the researchers suggest, is to monitor an agent's actions throughout an entire task, rather than merely focusing on its responses in isolated instances.