Apple is finally eliminating AI applications that can digitally strip individuals of their clothing.

      Apple has removed a number of AI-driven “nudify” apps from the App Store following legal pressure from the San Francisco city attorney. However, this situation raises a larger question beyond whether these specific apps breached App Store guidelines.

      The main concern is how these apps were allowed into one of the most closely regulated app marketplaces at all.

      This controversy erupted when San Francisco City Attorney David Chiu issued cease-and-desist letters to both Apple and Google, accusing them of enabling the distribution of apps that could create non-consensual AI-generated nude images. The legal notices indicate that both companies have profited from these apps through in-app purchases, leading California authorities to claim the platforms should cut ties with developers offering such services.

      In response, Apple confirmed that several of the flagged apps have been removed, while additional developers face permanent removal if they do not align with App Store policies.

      Apple stated these apps breach its regulations. Critics argue they should never have been available.

      Following WIRED’s investigation, Apple informed 9to5Mac that apps meant to create or spread pornographic or “nudification” content have always been against its App Review Guidelines. The company noted that three of the apps mentioned by the San Francisco City Attorney’s Office have been removed from the App Store, and the developer accounts associated with them are being terminated. Apple also stated it has reached out to four more developers, advising them to rectify policy violations or risk removal from the platform.

      In its statement, Apple said, “The App Store was designed to be a safe and trusted place for users,” emphasizing its proactive measures against nudification apps and its removal of others following user reports. Furthermore, Apple reiterated that developers are accountable for the content generated within their apps. Applications that entail user-generated content must have mechanisms to filter inappropriate material or face removal from the App Store entirely.

      Google has taken similar steps, indicating it has suspended hundreds of Android apps featuring nudification aspects and limited Play Store searches for terms like “nudify.” The company reported that the five Android apps mentioned by the San Francisco City Attorney have also been removed for violating Play Store policies.

      Apple provided 9to5Mac with a complete statement: “The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are contrary to our App Review Guidelines, and we have proactively rejected many of these apps while removing others, including those flagged by users via our reporting tools. We have removed three of the questioned apps and are in the process of terminating their developer accounts. We are communicating with four others who need to address policy violations or they may also face removal.”

      The broader challenge revolves around moderation in the AI age.

      The legal action is occurring amid rising concern regarding how generative AI has significantly reduced the barrier to creating explicit deepfakes. Unlike traditional pornography, many contemporary AI-powered face-swapping tools only require a single photograph. Users can swiftly generate fabricated nude images or explicit videos of individuals without their awareness or consent.

      Authorities assert that the repercussions extend beyond privacy breaches. Victims have increasingly reported harassment, extortion, reputational damage, and severe mental health issues. Previous investigations have also uncovered instances across numerous schools where AI-generated sexual images of minors were created using commercially available tools. Enforcement is particularly challenging because many of these applications do not openly market themselves as “nudify” tools; instead, they position themselves as benign AI face-swapping or photo-editing applications. The explicit image generation features are often concealed until after installation or are hidden behind in-app purchases, complicating automated moderation significantly.

      Research released earlier this year indicated that a substantial number of seemingly normal face-swapping apps could still produce explicit deepfakes despite not advertising themselves in that manner. In many instances, researchers discovered minimal or no safeguards against misuse.

      This differentiation has become central to the ongoing debate concerning Apple and Google.

      Both companies assert that their policies explicitly ban pornography and sexually exploitative content. However, critics contend that policies alone are insufficient when AI applications can mask harmful functions behind otherwise legitimate features. The City Attorney’s Office claims that Apple and Google also financially benefit from these apps through commissions on in-app purchases, stating that platform operators have a greater obligation to detect and eliminate these apps before they reach users.

      For Apple, the recent removals may address immediate criticism, but they are unlikely to conclude the larger discourse. As generative AI becomes more affordable, faster, and increasingly adept at creating realistic synthetic media, app review now involves more than merely verifying screenshots or developer descriptions. It requires assessing what an AI model might be coaxed into generating post-installation, a considerably more complex challenge than traditional moderation.

      This challenge is not expected to fade anytime soon, nor will the scrutiny directed at the companies overseeing the world’s largest app stores.