Medical AI substituted for nurses and evaded its own regulations.
The argument for medical AI is that it allows clinicians to focus more on patient care. However, two stories this week indicate that the situation might not be so straightforward. In New York, nurses claim they have been replaced by software. In Minnesota, a former leader at the Mayo Clinic warns that the software is not reliable.
Marilyn Shuler, who spent 39 years reviewing patient records at Montefiore hospital in the Bronx, was among a dozen nurses laid off this month, according to their union, due to the implementation of AI software that performs some of their responsibilities. The nurses were responsible for utilization review, which involved interpreting charts and negotiating with insurers regarding covered care. The New York State Nurses Association asserts that these layoffs violated a recently won contract through strike action, as reported by the Guardian.
Shaiju Kalathil, a nurse and union representative at Montefiore, expressed concern, stating, “It should also worry every practitioner and patient who values… the quality of care they receive.” In response, Montefiore refutes these claims, describing them as "inaccurate and misleading," and states that the change pertains to a nonclinical, paperwork system.
In the case of Minnesota, it raises issues of trust rather than employment. Traci Tamiko Eto joined the Mayo Clinic in 2023 to establish safeguards for its AI. In a recent lawsuit, she alleges that the hospital demoted and subsequently fired her after she raised concerns about the AI deployment, as reported by Minnesota Public Radio.
Her allegations are detailed. She claims the team responsible for an AI assistant named MAYA deleted unfavorable test results, exaggerated the tool's capabilities, and advanced it without appropriate oversight. According to her lawsuit, at one point, the tool had an error rate reaching as high as 67%. Mayo Clinic asserts its commitment to responsible AI, incorporating privacy, security, and compliance into its processes. However, if Eto’s account is validated in court, it indicates a system that bypassed the necessary checks for errors.
These two cases represent isolated incidents rather than a widespread trend, yet they highlight a shared concern in healthcare. While AI is expected to deliver distinct benefits in this field, its failures are particularly unacceptable. An ineffective advertisement may be embarrassing, but an inaccurate medical record can be hazardous.
The existing situation is already troubling. Utah has allowed AI to autonomously renew prescriptions without physician oversight. Research indicates that reliance on AI could gradually diminish professionals’ essential skills and shorten experienced careers. Companies like Allianz and Thomson Reuters are already replacing staff with AI models.
The central issue remains who bears the risk. Montefiore benefits from cost savings, while patients may suffer from what the software overlooks. Mayo Clinic gains efficiency, yet, according to the whistleblower, she was dismissed. Neither incident has been resolved in court, and both hospitals defend their practices. However, these stories challenge the neat narrative that medical AI is solely an assistance tool. Sometimes, it replaces human input, and often, those most knowledgeable about the technology are the first to be let go.
Other articles
X is instructing its AI algorithm on concepts that social networks used to grasp.
X is enhancing posts among mutual followers after discovering that its algorithm lacked relationship data, highlighting the discrepancy between forecasting engagement and fostering discussions that people truly desire.
Hassabis seeks a referee similar to FINRA for advanced AI developments.
Demis Hassabis advocates for a U.S.-led AI Standards Body, inspired by Wall Street’s FINRA, to evaluate frontier models prior to their release and to potentially slow industry progress if necessary.
I didn't fully appreciate this NotebookLM feature until it transformed my study habits entirely.
An upcoming exam prompted me to give NotebookLM's Mind Maps a try. Now, it's the first tool I turn to whenever I begin a new notebook.
The dream of the Concorde is attainable. The NASA X-59 simply needs to eliminate the supersonic boom.
NASA’s X-59 has achieved a speed of 924 mph at an altitude of 55,000 feet, moving the agency closer to demonstrating that future supersonic passenger flights can circumvent loud sonic booms.
Uber is advocating for the inclusion of robotaxis on its platform.
Uber's lobbying efforts in Washington, D.C., and New Jersey advocate for a "hybrid network" regulation that would allow driverless vehicles to operate alongside human drivers on the same platform.
Dimon states that AI has already removed 30 to 40 percent of jobs in certain divisions of JPMorgan.
Jamie Dimon informed analysts that AI has already removed 30 to 40 percent of jobs in certain JPMorgan divisions, but competitive pressures will diminish margin improvements.
Medical AI substituted for nurses and evaded its own regulations.
Two cases in the US highlight the negative aspects of medical AI: nurses at Montefiore being replaced by software, and a whistleblower from the Mayo Clinic claiming that a tool has a 67% error rate.