AI can outperform chess grandmasters, but it struggles to adjust to contemporary video games.

      Modern video games are highlighting the limitations of AI.

      Artificial intelligence is ubiquitous in our lives. Despite all the attention given to AI's achievements in games like chess and Go, as well as its forays into coding, there remains a notable shortcoming underlying these successes. AI struggles significantly with playing new video games that it has not encountered before.

      A recent paper from NYU argues that the significant milestones often touted in the media create a misleading impression of how close machines are to achieving true general intelligence.

      Understanding the distinctions is essential.

      While chess and Go are remarkable accomplishments, they are structured games with fixed rules, unlike the complex landscape of modern video games. According to NYU, AI still falls short of emulating human-like intelligence since it struggles to adapt.

      Where AI is deficient

      Researchers point out that many of AI's notable achievements in gaming stem from systems that are meticulously calibrated for a single game. Within those defined parameters, AI can appear superhuman. However, when there are even slight modifications to the rules or the environment, its outstanding performance may quickly falter.

      Artificial intelligence is everywhere. Jasmine Mannan / Digital Trends

      Video games serve as a genuine assessment of AI's intelligence. These games are not one-dimensional; they often require a wide array of abilities, including spatial reasoning, long-term planning, learning from experience, and even social intuition. The report suggests that this diversity makes video gaming a far better gauge of flexible intelligence than isolated benchmark tasks.

      Reinforcement learning and Large Language Models (LLMs) hit limitations

      The study further asserts that while reinforcement learning can yield remarkable outcomes, successful objectives are only reached after millions or billions of simulated trials. Thus, the system becomes adept only at the specific scenario it has been trained for. However, this proficiency breaks down when any changes occur. Even minor alterations, such as different colors or moved objects on a screen, can disrupt its functionality.

      LLMs do not offer a solution either. NYU reports that they tend to perform poorly on unfamiliar games. When they do succeed, it’s usually because of specific game-related frameworks that help them interpret game states, manage memory, and take actions. Remove that additional support, and their performance declines rapidly.

      The true benchmark

      The researchers contend that a genuine game-playing AI should be able to learn a new game from the ground up, in roughly the same amount of time it would take a skilled player—perhaps tens of hours—without extensive simulation or prior experience. This requirement exceeds the capabilities of current systems.

      This is significant beyond the realm of gaming. If AI struggles to adapt to an entirely new video game, it is even less equipped to manage the unpredictability of the real world. While chess might still generate headlines, modern video games reveal just how far AI still needs to progress.