Nvidia collaborates with chip competitor d-Matrix on inference technology.

      Nvidia has discovered a novel strategy to deal with its chip competitors: collaboration. The GPU leader is merging its hardware with inference chips developed by the startup d-Matrix, as reported by The Information. Together, they will deliver a joint system designed to operate AI models. The first customer for this system will be the AI-cloud firm Parasail, with deployment anticipated later this year. This approach aligns with Nvidia's broader tendency to partner with companies that aim to challenge its dominance.

      The rationale follows the structure of AI operations. Responding to a prompt involves two stages: a heavy "prefill" step ideal for Nvidia's GPUs, followed by a lighter, repetitive "decode" phase that can be performed more economically on specialized chips. Combining these two allows each to optimize its strengths.

      As for d-Matrix, it has designed its Corsair accelerator using “in-memory computing,” which places data close to the processing logic. Established in 2019, it secured $275 million at a $2 billion valuation in November and is currently seeking additional funding. CEO Sid Sheth asserts that its advantage lies in integrating compute and memory on a single chip. Furthermore, it avoids the costly, limited high-bandwidth memory that Nvidia's chips rely on. The startup contends that using Corsair alongside GPUs can process tokens approximately ten times faster at about one-third of the cost, consuming up to five times less energy for certain tasks.

      These figures come from the company itself rather than independent verification. However, they highlight the significant potential in the current AI landscape. While training focuses on developing models, inference—executing them for millions of users—represents where the operational expenses and revenue increasingly reside.

      Nvidia's strategy of assimilating competitors is notable because the firm usually does not require assistance. It leads in AI chip technology, and startups like d-Matrix aim to undermine that advantage. Instead of confronting each competitor, Nvidia continues to integrate rivals into its ecosystem, a method it has also employed with other chip and networking companies.

      This strategy mitigates risks. If specialized inference chips gain traction, like the custom silicon currently spreading in major tech, Nvidia stands to benefit from the transition rather than be adversely affected. Conversely, if such developments falter, it would face minimal losses. In any case, customers remain within Nvidia’s ecosystem.

      The significance of this partnership lies in its indication that the AI-chip market is evolving beyond a singular victor. Buyers are increasingly seeking the most cost-effective chip for each specific task rather than relying on one dominant GPU for all needs. Nvidia's response is to ensure that it either sells or collaborates with whatever emerges successful.