The social media network X has caused widespread outrage after people used its artificial intelligence assistant Grok to manipulate images of females by taking off their attire.
Numerous instances have emerged where the technology has been employed to digitally undress female subjects to make them appear in swimwear without permission, as well as to insert them in inappropriate situations.
Developer Reaction and Legal Stance
XAI, the organization that created Grok, offered no comment to requests for comment, other than an standard response stating "traditional news outlets lies".
A government official stated that legislation is being proposed to outlaw image stripping tools. Under a new law, anyone who distributed such technology would "face a jail term and heavy penalties".
"The platform could prevent these forms of misuse if they desired to," said a law professor, adding they "appear to enjoy impunity."
Regulator Position and Platform Capabilities
The regulator Ofcom said technology companies must "assess the danger" of people seeing illegal content on their networks, but stopped short of confirming whether it was presently looking into X or Grok in regarding artificial intelligence pictures.
Grok is a no-cost AI assistant—with some premium features—which answers X users' queries when they reference it in a message.
It is commonly used to provide reactions or additional background to other users' remarks, but people on X are also capable of alter an posted image through its AI image editing feature.
History of Criticism and Rules
The tool has been criticised for permitting users to produce photos and videos with explicit content and sexualised material, and it was earlier alleged to have been involved in making inappropriate material of celebrities.
"X has been permitting the creation and spread of these pictures for a long time without taking any measures and we have not witnessed any challenge by oversight bodies," Professor McGlynn added.
XAI's own usage rules forbids "depicting images of people in a pornographic fashion".
In a statement, Ofcom said it was unlawful to "generate or share deepfakes or illegal content" and confirmed this covered sexual deepfakes made with AI.
It said networks such as X were required to take "necessary steps" to "lessen the risk" of users encountering illegal material on their services and to take down it promptly when they learn of it.
