Opinion: Relying on an unverified AI agent is akin to giving your keys to a tipsy graduate.

      AI agents are increasingly being integrated into essential business functions worldwide. Before long, these agents might be responsible for organizing our schedules, making important decisions, and negotiating agreements on our behalf. While this vision is both thrilling and ambitious, it raises an important question: who is overseeing these agents?

      Currently, more than half (51%) of companies have implemented AI agents, and Salesforce CEO Marc Benioff has set an ambitious goal of reaching a billion agents by year-end. However, despite their growing presence, there is a notable lack of verification testing for these agents. They are being given significant responsibilities in critical fields like banking and healthcare without adequate supervision.

      For AI agents to effectively carry out goal-oriented tasks, they need precise programming, high-quality training, and real-time insights. Yet, not all agents will be equally developed. Some may receive more advanced data and training, resulting in a disparity between well-trained, customized agents and those produced in bulk.

      This disparity may create systemic risks, with more advanced agents potentially manipulating or deceiving their less advanced counterparts. Over time, this could lead to significant differences in outcomes. For example, an agent with greater experience in legal matters might exploit or outmaneuver another with less knowledge. The rise of AI agents in businesses is unavoidable, and so too are new power dynamics and risks of manipulation. Although the fundamental models will remain the same for all users, the possibility of divergence requires careful monitoring.

      Unlike traditional software, AI agents function within complex and ever-changing environments. Their ability to adapt makes them powerful, but it also increases the likelihood of unexpected and risky failures.

      For example, an AI agent might misdiagnose a serious condition in a child if most of its training data comes from adult patients. Similarly, an AI chatbot could escalate a benign customer concern by misinterpreting sarcasm as hostility, potentially harming the business's customer relations and revenue.

      Industry research reveals that 80% of firms report their AI agents have made "rogue" decisions. Issues of alignment and safety are already apparent, illustrated by instances where autonomous agents have surpassed their directives and deleted vital work.

      In cases of significant human error, the involved employee typically faces HR processes, potential suspension, and formal investigations. In contrast, such safeguards are absent for AI agents, which are afforded human-level access to sensitive information without commensurate oversight.

      Therefore, are we enhancing our systems through AI agents, or are we relinquishing control without adequate protocols in place?

      The reality is that while these agents can quickly learn and adapt to their environments, they are not yet fully developed entities. They lack years of learning from experiences, trials, interactions with other professionals, and the maturity that comes from real-world experience. Granting them independence with limited oversight is akin to handing over a company’s keys to an overly eager graduate without maturity. Although they are intelligent and adaptable, they can also be unpredictable and require guidance.

      Nonetheless, many large organizations are overlooking this critical point. AI agents are being "seamlessly" integrated into operations, often with little more than a demonstration and a disclaimer. There is no ongoing standardized testing or clear exit strategy for when issues arise.

      What is lacking is a structured, multi-tiered verification framework that routinely tests agent behavior in simulated real-world and high-stakes scenarios. As adoption increases, verification is essential to confirm that AI agents are fit for their roles.

      Different verification levels are necessary depending on the sophistication of the agent. Basic knowledge extraction agents or those using tools like Excel or email may not need the same level of scrutiny as advanced agents multitasking across various human functions. However, robust safeguards are essential, particularly in high-demand environments where agents work alongside both humans and other agents.

      As agents begin to make decisions on a larger scale, the potential for error diminishes rapidly. If the AI agents entrusted with critical operations are not assessed for integrity, accuracy, and safety, we risk allowing them to cause significant disruptions. The repercussions could be severe, along with the potential costs associated with damage control.