Apple is finally eliminating AI applications that are capable of digitally undressing individuals.

      Apple has removed a number of AI-driven “nudify” apps from the App Store after facing legal pressure from the San Francisco city attorney, but this incident raises a much larger issue than just the violation of App Store regulations by a few apps.

      The fundamental question is how these apps were allowed on one of the most tightly regulated app marketplaces globally in the first place.

      The recent controversy began when San Francisco City Attorney David Chiu sent cease-and-desist letters to both Apple and Google, accusing them of enabling the distribution of apps that can create non-consensual AI-generated nude images. The legal notices indicated that the companies have profited from these apps through in-app purchases, and California authorities argued that the platforms should cut ties with developers providing such services.

      Apple has responded by confirming that several of the identified apps have been removed, and other developers risk permanent removal if they do not adhere to App Store policies.

      Apple states that these apps violate its guidelines. Critics, on the other hand, argue that they should not have appeared in the first place.

      Following WIRED’s investigation, Apple informed 9to5Mac that apps designed to create or share pornographic or "nudification" content have always been in violation of its App Review Guidelines. The company noted that three of the apps mentioned by the San Francisco City Attorney's Office have been taken down from the App Store, and the developer accounts behind them are being terminated. Apple added that they have contacted four more developers, warning them to correct policy violations or face removal from the platform.

      "The App Store was designed to be a safe and trusted place for users," Apple stated, emphasizing that they proactively reject nudification apps and remove others based on user reports. Apple also reiterated that developers are responsible for the content generated within their applications, stressing that apps allowing user-generated content must implement mechanisms to filter inappropriate material or risk removal from the App Store entirely.

      Similarly, Google has acted by suspending hundreds of Android apps that offer nudification features and restricting Play Store searches for terms like “nudify.” The company reported that the five Android apps pointed out by the San Francisco City Attorney have also been removed for breaching Play Store policies.

      Here is Apple’s complete statement as provided to 9to5Mac: “The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. ‘Nudification’ apps are against our App Review Guidelines, and we have proactively rejected many of these apps and removed many others, including those flagged by users through our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts. We are in contact with four others that need to address policy violations or risk removal as well.”

      The larger challenge lies in moderation in the age of AI.

      The legal actions come amidst rising concerns over how generative AI has significantly lowered the barrier to creating explicit deepfakes. Unlike traditional pornography, many contemporary AI-driven face-swapping tools require just a single photo. With a few clicks, users can create fake nude images or explicit videos depicting real individuals without their consent or awareness.

      Authorities assert that the repercussions go beyond privacy breaches. Victims have increasingly reported issues like harassment, extortion, reputational harm, and severe mental health effects. Previous studies have also revealed incidents in numerous schools where AI-generated sexual images of minors were produced using readily available tools. The enforcement of regulations is particularly challenging since many of these applications do not openly advertise themselves as “nudify” tools, instead presenting themselves as innocuous AI face-swapping or photo-editing apps. The abilities to generate explicit images often remain concealed until after the app is installed or behind in-app purchases, complicating automated moderation significantly.

      Research conducted earlier this year indicated that numerous seemingly benign face-swapping apps could still produce explicit deepfakes, despite not marketing themselves in that manner. In many instances, researchers found minimal or no protections against misuse.

      This distinction has become crucial to the current debate surrounding Apple and Google.

      Both companies maintain that their policies explicitly ban pornography and sexually exploitative content. Critics, however, argue that policies alone are insufficient when AI applications can mask harmful functions behind otherwise legitimate features. The City Attorney’s Office further claims that Apple and Google financially benefit from these apps through commissions on in-app purchases, stating that platform operators have a greater obligation to identify and eliminate them before they can reach users.

      For Apple, these latest removals may help mitigate immediate criticism, but they are unlikely to halt the larger discussion. As generative AI becomes cheaper, faster, and increasingly adept at producing realistic synthetic media, app review processes are no longer merely about scrutinizing screenshots or developer descriptions. They now require the ability to identify what an AI model could be coaxed into generating after installation, a significantly more complex endeavor than traditional moderation.

      This challenge is not expected to resolve soon, nor will the questions directed at the companies managing the world