Utah allowed artificial intelligence to prescribe medication.

      The argument for AI in prescription renewals is compelling, but so is the caution against relying on a state sandbox to mitigate associated risks. In January, a security research firm named Mindgard engaged with a chatbot developed by Doctronic, a health tech startup that recently became the first company in U.S. history to receive state approval for autonomous medical prescription renewals using AI. Mindgard’s team provided the AI with a fabricated regulatory bulletin and observed its response. The AI, misled by a non-existent document, claimed it would increase the standard dose of OxyContin threefold.

      Doctronic and Utah’s Office of Artificial Intelligence Policy quickly emphasized that the compromised chatbot was their public tool, not the secured system overseeing the prescription pilot. This distinction is essential and should be taken seriously, but it does not address the fundamental concern raised: whether a 12-month state sandbox program managed by a commerce department aiming to promote AI innovation is an appropriate method for addressing such questions.

      First, it’s important to understand the genuine issue Utah is attempting to tackle. For many Americans, prescription renewals represent a bureaucratic hurdle that has no clinical benefit. According to the CDC, nearly half of all individuals with chronic conditions do not adhere to their prescribed medications. The broader challenge of making healthcare more accessible and preventative rather than reactive is something the technology sector has been addressing for years.

      A significant portion of medication non-adherence is tied directly to the renewal process: the two-week wait for a primary care appointment, missed calls from the office, and expired prescriptions that necessitate starting over. Managed Healthcare Executive reported that Doctronic’s co-founder Matt Pavelle estimated this figure to be around 30% of all non-adherence.

      This is a substantial figure linked to a tangible and resolvable issue. Medication non-adherence costs the American healthcare system between $100 billion and $300 billion annually, depending on various studies, and is associated with approximately 125,000 preventable deaths each year. These figures are based on peer-reviewed research and CDC data, not just a startup’s promotional materials.

      Thus, the argument for AI in prescription renewals is significant, particularly where healthcare access is limited: in rural areas, among low-income patients, and for elderly Americans who find it difficult to attend in-person appointments. Doctronic’s co-founder, vascular surgeon Adam Oskowitz, clearly stated in January that patients wait weeks to renew prescriptions for medications they have been using for years, for conditions that remain unchanged. This delay is not a system feature but a failure. If AI can address this failure safely, it should be employed.

      The critical issue lies in the term “safely.” Doctronic claims that its AI aligns with human clinicians’ treatment plans 99.2% of the time across 500 urgent-care scenarios. The company presented these statistics to Utah regulators, which seemed persuasive.

      However, 500 cases is a limited dataset for a system that will ultimately handle prescriptions on a larger scale. The 0.8% that did not match translates, at significant volumes, to a substantial number of patients receiving alternatives to what a clinician would have suggested. Furthermore, matching recommendations in a structured evaluation does not guarantee robustness against all real-world scenarios, including potential adversarial challenges.

      The Mindgard experiment was not a stress test of the active system; it demonstrated that the publicly accessible AI could be manipulated with a forged press release. While the distinction between the live system and the public tool is reassuring, it does not provide conclusive evidence of safety.

      What warrants careful evaluation of the Utah arrangement is its regulatory framework. The state’s Office of Artificial Intelligence Policy, established in 2024, can exempt companies in its regulatory sandbox from its unprofessional conduct laws, as it did for Doctronic. The pilot has three phases, starting with physician review of all renewals, which seems thorough. In phase three, only 5% to 10% of renewals are reviewed by physicians, while the remainder occurs autonomously. STAT News raised the issue of whether an AI system that assesses clinical data and issues prescriptions should be classified as a medical device regulated by the FDA.

      That question remains unresolved. Utah lacks the authority to address it, and its agreement with Doctronic does not necessitate FDA approval before the system expands.

      Both the American Medical Association and the Utah Academy of Family Physicians voiced formal objections. Dr. John Whyte, CEO of the AMA, stated that excluding physicians from clinical decisions endangers patients. The Utah Academy expressed concern that the program seems willing to advance AI without adequate safety measures.

      These professional associations may have vested interests, as they could be impacted by AI reducing demand for their services, but concerns regarding safety measures are distinct from these interests. A state commerce department has different incentives compared to a regulator focused primarily on patient safety.

      Utah’s OAIP is explicitly tasked with promoting AI adoption, which is a valid policy aim but should not dominate the evaluation of prescription safety