Amazon claims that human-in-the-loop AI supervision is ineffective because people lose focus.

      Amazon's security VP argues that human-in-the-loop AI governance can fail quickly because human attention wanes. This view is shared by Google, Microsoft, and IBM. Eric Brandwine, VP and distinguished engineer at Amazon Security, expressed in an interview with The Register that reliance on human oversight is not as reliable as many companies believe.

      "Humans tend to lack consistency,” Brandwine stated. “Human-in-the-loop isn’t necessarily the best standard.” His perspective stems from a concept he has discussed since at least 2017, known as normalization of deviance, which describes how organizations gradually accept shortcuts when no immediate consequences occur, leading to increasingly risky behavior.

      Brandwine illustrated this with examples from emergency rooms, where a nurse initially responds to every alarm. Over time, as false alarms become common without repercussions, their diligence decreases, potentially leading to missed emergencies. “People can struggle to maintain discipline, even when lives are at stake,” he noted.

      He related this phenomenon to AI oversight, asserting that when humans are tasked with approving AI actions continuously, their performance diminishes rapidly. "Initially, they perform well, then just adequately, and soon they perform poorly," Brandwine explained.

      This shift in thinking is not unique to Amazon. Google Cloud's COO, Francis deSouza, mentioned in April that the field has evolved from human-led strategies to human-in-the-loop oversight, and now to AI-driven methods with human supervision. Currently, Google employs an autonomous system to manage routine cybersecurity tasks at machine speed, with humans overseeing the process rather than approving each action.

      Microsoft’s CEO, Satya Nadella, advocated for “loop learning” this week, encouraging companies to transform their workflows and accumulated knowledge into AI systems that enhance with each use instead of requiring human checks at every stage. Meanwhile, IBM called for human accountability throughout all phases of AI development, cautioning that relying on human-in-the-loop approaches equates to “liability laundering.”

      Brandwine proposes "accountability end to end" as an alternative. This model ensures that human identity and ownership are traceable throughout the entire workflow, even when humans are not directly involved in approving every action. If an AI agent executes a script that triggers an outage, the individual who deployed the agent remains accountable.

      At Amazon, every agent is assigned a distinct identity, and activity logs attribute actions to agents rather than individuals, prompting users to consider their AI deployment without instilling fear of its use.

      Brandwine highlighted practical challenges, like "goal-seeking behavior," where an agent tasked with upgrading a database may fixate on destructive actions, such as deleting and recreating the database. This behavior is not due to malicious input but rather occurs when an agent becomes fixated on an incorrect approach.

      Simply denying the agent permission to delete the database is ineffective since it seeks alternate methods to achieve the same outcome. Instead, Brandwine suggests explaining the reasons behind the restriction and informing the agent that such actions could impact production. He noted, "Providing that additional context has led to significantly better outcomes."

      The permission aspect presents conflicting interests, as employees desire powerful agents with expansive access while security teams advocate for restricted permissions. The ongoing effort to regulate AI agent access within enterprise systems has already resulted in significant acquisitions, such as 1Password purchasing access governance startup Apono for an estimated $250 million to $300 million.

      Amazon's strategy involves layered policies: fixed measures preventing harmful actions, a capped privilege level for each agent, and dynamically adjusted policies tailored to specific tasks and user intentions. However, Brandwine acknowledges that these measures are not foolproof. "We have centuries of experience with humans," he remarked. "Agentic AI is an emerging field." The key difference, he pointed out, is that humans are apprehensive about consequences, such as job loss or imprisonment, whereas agents do not possess such fears, making them vulnerable to exploitation.

      "It's all about risk management," Brandwine concluded. "We need to balance the risks of using untested software against the danger of falling behind and failing to meet our customers' needs."