Nvidia is assembling a team focused on AI safety, and there is a business rationale behind it.

      Nvidia's philosophy can be seen in a recent job advertisement. The chipmaker is forming a new AI safety and security engineering team, according to Business Insider. A batch of job postings from late last month reveals this initiative. The message conveyed is not one of caution regarding AI's potential risks, but rather a belief that trust is essential for encouraging its adoption.

      The team being assembled is compact but highly experienced. Nvidia is seeking a notable engineer to serve as its “founding technical leader” along with other roles, including a security research engineer, an evaluation engineer, and a senior manager. Their responsibilities, according to the postings, will involve assessing AI agents prior to their deployment and developing AI-driven tools to address software vulnerabilities.

      The case for openness

      One of the job postings clearly states this belief. It identifies open-weight models, transparency, and extensive scientific examination as “fundamental to American AI leadership and cybersecurity”. Open-weight models disclose the trained weights that determine a model's behavior while keeping training data and source code confidential. This stands in contrast to the closed systems created by companies like OpenAI and Anthropic.

      Nvidia has been vocal about this stance. In his inaugural post on X last month, CEO Jensen Huang shared a letter encouraging US lawmakers to support open models, asserting that they “enhance safety and cybersecurity”. Shortly afterward, Nvidia became a founding member of the Open Secure AI Alliance.

      This alliance includes 120 companies such as Microsoft, Palantir, and SpaceX, and is focused on developing open-source security solutions. The listings for the safety team surfaced just ahead of this initiative and outline similar objectives.

      The reason safety drives chip sales

      The philosophy has economic reasoning behind it. Open models allow AI to reach a broader audience compared to a closed service. Each new user requires hardware to operate it, which is why competitors like AMD are pursuing the same open-source opportunity. Nvidia aims to maintain its position in this space. Increased usage translates to higher demand for chips.

      Safety is crucial for facilitating this adoption. As AI evolves from chatbots to agents that can access sensitive organizational data and act on it, trust becomes the main concern, overshadowing mere capability. Companies are unlikely to grant an autonomous agent access to their systems unless they are confident in its safety.

      This is why organizations are now assessing agents for unauthorized behaviors prior to activating them.

      The debate around open models remains unresolved. Critics argue that releasing weights could empower malicious actors just as easily as it aids defenders. Researchers have highlighted a safety gap, indicating that while open models can match the leading edge in capability, they lag behind in safety measures.

      Nvidia did not respond to Business Insider’s request for comments, so its perspective is primarily reflected in the job advertisements. It’s important to acknowledge who stands to gain if this argument prevails; the company advocating for safe, open AI is also the one that sells the necessary hardware.