AI is capable of defeating chess grandmasters, however, it struggles to adjust to contemporary video games.

      Modern video games reveal the limitations of AI capabilities.

      Artificial intelligence is pervasive in our lives.

      Despite the attention surrounding AI's achievements in chess, Go, and even coding, a significant weakness remains evident beneath these successes. AI struggles significantly when faced with a new video game that it has never encountered before.

      A recent paper from NYU emphasizes that these headline-making milestones have created a distorted view of how close machines are to achieving true general intelligence.

      The distinction is important.

      While feats in chess and Go are remarkable, these games operate with fixed rules and structured settings, unlike the intricate modern video games. NYU points out that AI has not yet achieved human-like intelligence as it struggles with adaptability.

      Where AI falls short

      Researchers note that many of AI’s notable achievements in gaming stem from systems fine-tuned to a single game. Within those specific parameters, AI can perform at a superhuman level. However, even minor alterations to the rules or environment can lead to a significant decline in its performance.

      This is where video games serve as a genuine litmus test for AI intelligence. Games typically demand a diverse array of skills, including spatial reasoning, long-term planning, trial-and-error learning, and social intuition. According to the report, this diversity makes gaming a much better indicator of flexible intelligence than isolated benchmark tests.

      Reinforcement learning and LLMs encounter limitations

      The research paper indicates that while reinforcement learning can yield impressive results, it typically requires millions or billions of simulated runs to achieve acceptable outcomes. As a result, the system excels only in the exact scenarios it was trained for. However, this expertise falters when any modifications are introduced. Even simple changes, like altered colors or repositioned objects on a screen, can disrupt its performance.

      Large Language Models (LLMs) do not resolve this issue either. NYU notes that they perform surprisingly poorly in unfamiliar games. When they do manage to perform well, it's usually due to custom game-specific frameworks designed to interpret game states, manage memory, and execute actions. Remove that additional support, and their performance declines sharply.

      The true benchmark

      The researchers posit that a genuinely effective game-playing AI would need to learn a new game from the ground up in approximately the same amount of time as a skilled player—perhaps tens of hours—without relying on extensive simulation or prior knowledge. All of this is beyond the capabilities of current AI systems.

      This has broader implications beyond gaming. If AI struggles to adapt to a brand-new video game, it is even less equipped to handle the unpredictability of the real world. While chess may still generate headlines, modern video games highlight the significant distance AI still has to cover.