How to stop restraining AI agents

      Developers of agentic AI have made ambitious claims about creating autonomous systems capable of handling various tasks, such as booking flights, monitoring competitors in real time, and managing entire procurement cycles, all without human input for confirmations. Although much of the required technology is already available, the necessary infrastructure to implement it reliably at scale has significant shortcomings.

      Gartner recently estimated that more than 40% of agentic AI initiatives might be halted before 2027, citing rising costs, ambiguous business value, and insufficient risk management as key reasons. This is quite notable, especially against the backdrop of expectations that autonomous agents would signify a major advancement in AI. However, this outcome shouldn’t surprise anyone familiar with the clear limitations these agents display in real-world applications. Many assume that the primary issue lies in the quality of the models, but this view is somewhat misguided.

      Understanding the Web's Resistance to Agents

      To appreciate what a capable agent truly needs, one must recognize that simply accessing a website and receiving a response is just the beginning. The agent must also convert that response into a useful format. Moreover, it needs to perform this consistently, in real time, and at a scale that justifies the effort.

      Given the current state of the web, this task is quite challenging. For instance, online platforms could technically allow an independent agent to assess and select the best platform based on user preferences. Yet, these platforms rely on information being less accessible to preserve their competitive edge. They implement increasingly personalized results, sponsored placements, and urgency signals to influence user behavior in their favor. Without access to relevant data, no AI agent can successfully navigate the web or automate the selection of the best options for users.

      Consequently, while the web functions reasonably well for general browsing, it systematically undermines automated access. Some forthcoming findings provide a clear view of this situation.

      Oxylabs is set to launch the Web Openness Index, which evaluates over 120 countries based on various web accessibility factors. The results indicate:

      - A global average score of 83.4 out of 100 for practical reachability, meaning how effectively a site responds to standard automated HTTP requests.

      - An average score of 62.8 for anti-automation friction (where lower scores indicate greater friction), which includes CAPTCHAs, rate limiting, fingerprinting, and bot detection.

      - And a further decrease to 60.3 in terms of structured data interoperability, reflecting whether websites provide data in machine-readable formats.

      The more than 20-point discrepancies highlight a structural gap. While sites generally react to automation requests, numerous restrictions are in place, and data is frequently presented in formats that are not machine-friendly. Consequently, agents that rely on timely, reliable, and structured information often fall into this gap.

      AI Starved for Data

      Within organizations, agents encounter a related issue: a shortage of usable data. Although the relevant data may exist, it often lacks the necessary cleaning, tagging, or structuring for an AI system to comprehend.

      This issue also extends to customer-facing applications developed on agentic systems. Without real-time web data, such as current prices, live inventory, policy changes, and market trends, these applications can only operate based on outdated information.

      Latency presents another challenge. An agent that eventually provides the correct answer is significantly less valuable than one that can deliver answers swiftly enough to take action. The threshold for acceptable delays is even lower in the context of autonomous systems.

      In all instances, the fundamental constraint remains the same: agents require trustworthy context, which they are currently not receiving, neither from their organizational data nor from the web.

      Addressing a Previously Resolved Issue

      It's important to remember that this is not the first occasion in which the volume of information has surpassed our ability to interpret it. The early web serves as an instructive example. While it held a wealth of knowledge, that information was not useful in its unrefined form. The key difference then was the establishment of scalable infrastructure. Web crawlers were deployed to index websites, scrapers were utilized to compare online prices, and monitoring systems tracked fraudulent advertisements and brand impersonation across many domains. All these innovations necessitated the reliable and scalable collection of publicly available web data.

      A more recent case comes from our pro bono Project 4β partners, Debunk.org. This nonprofit organization, dedicated to combating online disinformation and fraud, uncovered a large-scale, multilingual scam operation targeting previous victims of fraud. Their investigation identified over 50,000 advertisements, 459 domains, and over 1,100 associated web pages, reaching an estimated 52 million individuals across Europe. Such extensive coverage demands systematic and automated data collection at scale.

      Agentic AI requires similar infrastructure but with even greater demands since these agents utilize data in ways that surpass past applications. They need information that is structured, up-to-date, complete, and delivered quickly enough to facilitate real-time actions.

      The Three Cs of Reliable Agent Infrastructure

      As previously mentioned, achieving this infrastructure is unlikely