What lies ahead for AI and the steps to achieve it

      **Summary**: Frontier AI models have nearly aligned, with a recent Stanford report indicating a convergence within 5%. Gartner forecasts that over 40% of agentic AI projects will be terminated by the end of 2027 due to rising costs and unclear business benefits. Oxylabs' SVP Gediminas Rickevičius emphasizes that the critical aspect has evolved from selecting models to establishing robust data infrastructure, focusing on structured web indexes for agents and real-time information access. McKinsey reveals that 88% of organizations utilize AI, but only 6% are classified as high performers.

      In 2026, the discourse around AI is changing significantly. Two years ago, boardroom discussions revolved around which AI model to adopt. Now, that conversation has less significance as leading models have converged—Stanford's 2026 AI Index notes significant performance gains, with top models clustering closely in effectiveness. The model itself is no longer the central focus; attention has shifted elsewhere.

      That shift pertains to data—specifically, what models access, when they access it, and its structuring quality. Organizations face two primary obstacles: one, the problem of AI agents making confident errors; and two, the foundational challenge of acquiring fresh information that necessitates a new kind of search infrastructure. Both issues ultimately trace back to data.

      **Future Directions: AI agents and the agentic web**

      **The Failures of Agents**

      Agentic AI—systems capable of planning, data searching, tool usage, and executing complex tasks with minimal human input—are being used for various applications such as monitoring, pricing intelligence, market research, and lead qualification. Unfortunately, these agents often underperform.

      Gartner foresees that more than 40% of agentic AI initiatives will be scrapped by the close of 2027, attributing this to rising costs, undefined business value, and insufficient risk management. A 2025 MIT NANDA study starkly reports that nearly 95% of generative AI trials have failed to yield measurable results, primarily due to fragile workflows and a lack of contextual learning.

      When an agent produces a confident error, the instinct is to attribute the blame to the model—whether it be a reasoning flaw or a hallucination. However, these failures can frequently be traced to simpler issues: the agent was operating with outdated or incomplete information. In other words, it was working with stale facts. An agent evaluating a competitor's pricing strategies is constrained by the availability of current and geographically relevant pricing data, not by its reasoning capabilities. Such agents employed across crucial business operations pose a structural risk that no level of prompt optimization can remedy.

      **Context as Infrastructure**

      Within enterprise AI budgeting, there is a recurring trend to consider data acquisition as a low-priority expense, something beneath the surface and not revisited in strategic discussions. This mindset is becoming increasingly untenable. The industry has even coined the term "context engineering" to address this challenge, with Anthropic defining it in late 2025 as the curation of the ideal information set available to a model during inference.

      For organizations relying on web data—like product listings, financial disclosures, news updates, job postings, regulatory changes, and competitor activity—the raw data exists. The challenge lies in consistently accessing it. Issues such as dynamic content rendering and inconsistent data formats hinder the quick and scalable extraction of public data. Organizations that fail to adequately invest in overcoming these challenges allow convenience to dictate their AI's worldview.

      **Enhanced Web Indexes**

      To provide agents with up-to-date information, a robust web index is essential. This is an organized database that search engines use instead of the live internet, functioning similarly to a library catalog, allowing quick information retrieval from billions of pages without needing to extensively read each one.

      The indexing process is ongoing, with crawlers finding new pages, parsers filtering out unnecessary elements like advertisements and code, and the system saving clean content with quality metrics for ranking. While traditional indexes offer links for users to follow, AI agents require structured data for immediate use. Conventional indexes force agents to retrieve, clean, and summarize web pages individually, resulting in delays and increased token costs. Agent-specific indexes eliminate this need by storing ready-to-use content with verifiable sources, transforming search from a list of links into a reliable actionable resource.

      The distinction becomes apparent for those involved in AI development: traditional search indexes were designed for human users, presenting titles, links, and brief snippets sufficient for choosing a link to click. In contrast, an agent needs the actual information, pre-extracted and structured for language model integration.

      When an index only provides links, the agent must retrieve each page, extract the relevant information, and summarize it, causing delays, escalating token expenses, and potential breakdown points. Agent-centric indexes help avoid this scramble by storing content in a directly usable format, including source and date, enabling fact verification. This aspect is crucial, as demonstrating the origin of each claim serves as