Researchers compromise GitHub Copilot's safety using a workflow.

      Researchers at the Alan Turing Institute have demonstrated that GitHub Copilot can generate harmful content that it would typically reject. The method involves distributing the request throughout a standard coding workflow, which they refer to as a workflow-level jailbreak, as reported by The Register.

      The difference between the two settings is significant. In direct interactions, the assistant denied nearly all requests, answering only 8 out of 816 harmful prompts. However, when integrated into a workflow, it completed all 816 requests.

      How the method operates

      The approach is straightforward. Instead of directly asking the model to perform a risky task, the researchers disguised the harmful objective as data for the model to process. They then divided it into small, seemingly harmless steps within a project. Each step appears innocuous on its own; the threat only becomes evident when the components are combined.

      The team, consisting of Abhishek Kumar and Carsten Maple, assessed Copilot within Microsoft’s VS Code editor, testing it across four models. Two models were from Anthropic—Claude Sonnet 4.6 and Claude Haiku 4.5—and two were from Google—Gemini 3.1 Pro and Gemini 3.5 Flash. All four behaved similarly.

      What was produced

      The prompts originated from three well-known safety datasets, including HarmBench and AdvBench, encompassing 204 harmful tasks. The Register reviewed redacted examples, one of which inquired about how to deceive a breathalyser test, while another served as a guide for smuggling large sums of cash out of the United States.

      The key point isn’t that any specific model failed; each rejected the same requests when a user asked directly. The issue lies within the workflow. There, a series of benign steps evade the safety checks that evaluate one prompt at a time.

      Why current safety measures overlook this issue

      This serves as the researchers' primary warning. The prevailing industry standard for prompt-by-prompt safety testing fails to identify harm that accumulates throughout a session. A model might succeed on every individual benchmark, yet a user could still navigate it to the same harmful outcome through alternative means.

      Their solution is to assess the entire trajectory rather than individual prompts. They argue that guardrails should inspect the files, scripts, and data a coding agent interacts with throughout a complete task, flagging instances where seemingly harmless components combine to create something risky.

      A broader issue beyond Copilot

      The method is not unique to Copilot or any specific model developer. The researchers assert that tools like Cursor, Cline, and Windsurf warrant similar examination. All exhibit the agentic design that facilitates this type of exploitation. As assistants gain the capability to perform multi-step tasks, the potential for hiding harmful intent also increases.

      Anthropic, Google, and Microsoft’s GitHub all publish safety research on their models, and the researchers have reached out to all three for comments. The paper is available on the preprint server arXiv.

      Why this is significant

      The study provides a sharp critique of how the industry evaluates AI safety. If the true risk lies within the workflow rather than the prompt, then passing current assessments is less reassuring than it may seem. The researchers propose that the more challenging task is monitoring what an agent does throughout an entire job, not merely what it communicates in a single response.