Medical AI has supplanted nurses and evaded its own regulatory checks.
The argument for medical AI is that it allows clinicians to focus more on patient care. However, two recent incidents indicate a different reality. In New York, nurses claim they have been replaced by software, while a former leader from the Mayo Clinic in Minnesota asserts that the software is not reliable.
Marilyn Shuler dedicated 39 years to reviewing patient charts at Montefiore hospital in the Bronx. This month, she was among a dozen nurses laid off, having been replaced by AI software that performs part of their tasks, according to their union. The nurses were responsible for utilization review, which involved examining charts and negotiating with insurers about covered care. The New York State Nurses Association argues that the layoffs violated a contract recently secured through a strike, as reported by the Guardian.
“It should also concern every practitioner and patient who cares about... the quality of care they receive,” stated Shaiju Kalathil, a nurse at Montefiore and union representative. Montefiore contests this narrative, labeling the union's claims as “inaccurate and misleading” and claiming that the changes pertain to a nonclinical paperwork program.
If the situation in New York involves job loss, Minnesota's scenario relates to trust, and it is more troubling. Traci Tamiko Eto joined the Mayo Clinic in 2023 to help create safeguards for its AI. In a new lawsuit, she alleges that the hospital demoted her and subsequently terminated her for raising concerns about the AI implementation, as reported by Minnesota Public Radio.
Her allegations are detailed. She claims that the group behind an AI assistant named MAYA deleted unfavorable test results, exaggerated the tool’s capabilities, and advanced it without adequate oversight. According to the lawsuit, the tool had an error rate as high as 67% at one point.
Mayo asserts its commitment to responsible AI, with privacy, security, and compliance as central elements of its practices. However, if Eto’s account is validated in court, it reveals a system hastily moved past necessary checks to identify errors.
These two instances represent isolated cases rather than a broader trend, yet they reflect a shared underlying issue. Healthcare was anticipated to showcase the clearest benefits of AI, making its failures particularly unacceptable. An erroneous advertisement may be embarrassing, but an incorrect chart poses serious risks.
The broader situation is already concerning. In Utah, AI has been allowed to renew prescriptions without a doctor’s involvement. Studies continue to show that relying on AI can subtly diminish the skills that professionals are expected to maintain, shortening career paths for experienced individuals. Companies like Allianz and Thomson Reuters have begun replacing employees with AI models.
The common thread throughout these cases is the distribution of risk. Montefiore benefits financially from the layoffs; patients face the consequences of any software errors. Mayo gains efficiency, while, according to the whistleblower, she lost her job.
Neither situation has been adjudicated in court, and both hospitals defend their practices. Yet, together they challenge the tidy narrative that medical AI serves only to provide assistance. At times, it also eliminates human roles. Often, the individual most knowledgeable about the machine is the first to be dismissed.
Other articles
Hassabis seeks a referee similar to FINRA for advanced AI developments.
Demis Hassabis advocates for a U.S.-led AI Standards Body, inspired by Wall Street’s FINRA, to evaluate frontier models prior to their release and to potentially slow industry progress if necessary.
Microsoft's emissions increased by 25% due to the expansion of AI data centers.
Microsoft's emissions increased by approximately 25% in 2025, reaching 20.3 million tonnes, largely due to AI data centres, making its goal of being carbon-negative by 2030 more challenging.
Dimon states that AI has already reduced jobs by 30 to 40 percent in certain divisions of JPMorgan.
Jamie Dimon informed analysts that AI has already reduced employment by 30 to 40 percent in certain JPMorgan departments, but competitive forces will diminish margin improvements.
Hassabis seeks a referee comparable to FINRA for advanced AI.
Demis Hassabis advocates for a US-led AI Standards Body, inspired by Wall Street’s FINRA, to evaluate frontier models prior to their launch and to potentially decelerate the industry if necessary.
Medical AI substituted for nurses and evaded its own regulations.
Two cases in the US highlight the negative aspects of medical AI: nurses at Montefiore being replaced by software, and a whistleblower from the Mayo Clinic claiming that a tool has a 67% error rate.
I didn't fully appreciate this NotebookLM feature until it transformed my studying approach completely.
An upcoming exam finally motivated me to give NotebookLM's Mind Maps a try. Now, it's the initial feature I utilize whenever I create a new notebook.
Medical AI has supplanted nurses and evaded its own regulatory checks.
Two cases in the US highlight the negative aspects of medical AI: nurses at Montefiore were substituted by software, and a whistleblower from Mayo Clinic claims that a tool has a 67% error rate.