In dark corners of the internet, depraved content creators are sharing pornographic pictures and videos made with artificial intelligence to make money - and humiliate their loved ones.
Four years after police concluded a massive bust of depraved content creators in South Korea, deepfake technology brings a sinister new dimension to channels, with hundreds of thousands of people signed up to view digitally altered pictures of women and children.
Despite efforts to crack down on distributors of illegal porn online, users also continue to share photos and videos of strangers, family members and schoolchildren candid and unaware, drugged or sleep, in compromising positions.
Investigations to infiltrate and take down these rooms have uncovered horrifying hazing rituals, with young people even sharing indecent images of their own family members to gain access.
South Korea is battling an online scourge that continues to grow in spite of lengthy sentences and high-profile cases. But with the technology now available world over, the nefarious case of the 'New Nth Room' carries stark implications for us all.
The horror story of similar chatrooms operated 2018-2020 became the focus of a Netflix documentary, Cyber Hell: Exposing An Internet Horror (pictured)
Cho Ju-bin, nicknamed 'Doctor', was only named by police in 2020 after five million people, including local celebrities, signed a petition for him to be identified
One deepfake generating bot tells users: 'Hello sir, I am the magic photo bot. The woman in the picture... well try it and you'll see. Let's start sending pictures of the girl you like right now!' On the right, another post reads: 'Stripped down... invite a friend to process your photos'.
In the original Nth room case, at least 103 people, including minors, were confirmed as victims
In recent weeks, much has been made of the online chatrooms in which users are building communities around the distribution of pornographic pictures.
Investigative journalists are calling the trend the 'New Nth Room', in reference to a similar case that plagued citizens between 2018 and 2020.
Read More
AI is being used to generate deepfake child sexual abuse images of real-life victims, report warns
A virulent string of sex crimes were then exposed by South Korean journalist Kim Wan, who was investigating a student allegedly distributing child pornography through Telegram.
The 'Nth Room case', as it came to be known, saw at least 103 people, including 26 minors, confirmed as victims of a huge underground market for pornographic media sold with cryptocurrency on the messaging platform.
It later emerged that victims were being blackmailed and extorted over the media traded across various channels and groups on Telegram.
With national attention falling on the case, high-profile creators received lengthy jail sentences for their involvement, possibly to deter would-be copycats.
Cho Ju-bin, nicknamed 'Doctor', was only named by police in 2020 after five million people, including local celebrities, signed a petition for him to be identified.
Police said Cho, then 25, threatened at least 74 teens and young women to film explicit videos and used illegal videos of famous faces - including underage celebrities - to bait users into paying to join a Telegram chatroom.
Cho was found guilty that November of running a group which blackmailed girls into sharing sexual videos and sentenced to 40 years in jail.
He apologised 'to those who were hurt by me' and appeared to thank investigators, saying: 'Thank you for putting a brake on the life of a devil that could not be stopped.'
'Doctor' Cho was reported to have been the mastermind behind the operation.
Soon enough, others followed.
Moon Hyung-wook, then 24, was arrested in May 2020 and sentenced to 34 years in prison for 'coercing dozens of women and girls into sharing sexually explicit videos of themselves'.
Moon, known by his online username 'GodGod', was indicted on 12 counts, including violating laws on protecting minors from abuse.
The horror story later became the focus of a Netflix documentary, Cyber Hell: Exposing An Internet Horror.
Cho Ju-bin, known as the 'Doctor', was sentenced to 40 years for leading the online sex ring
Moon Hyung-wook, known by his online username 'GodGod', was sentenced to 34 years
Despite the public arrests and strong sentences, South Korean outlets suggest we may be witnessing a revival of a similar trend - now assisted by use of AI.
Koreaboo reported recently that some of the groups require new members to pass a kind of hazing test by sharing illicit photos of their mothers, sisters, and other women in their lives - 'and provide the girl's personal details'.
Read More
Giorgia Meloni tells court deepfake porn videos of her are 'a form of violence against women'
'Some of these rooms revolve around 'humiliating' one's family members by taking photos of videos of oneself inappropriately touching one's sister or mother,' the outlet reported.
Horrifically, the images reportedly extend to the distribution of pictures of minors.
Some women are asleep or drugged, according to the Malay Mail. Some users are pictured 'acting inappropriately' with the victims, including mothers, siblings or relatives'.
'Most victims were middle and high school students,' Koreaboo claimed.
'When one participant hesitated to abuse an elementary school student born in 2013, others criticised him, saying: 'Her age is double digits. What's the problem?''
The outlet, which has been central in helping to identify and expose the groups online, reported that 'various sources and bokep chindo victims' have uncovered rooms with a total of more than 212,000 participants.
Twitter/X user Queenarchive1 brought the issue to light with a series of posts of Telegram rooms.
A screenshot of one of the rooms reportedly showed one of the users lifting their younger sister's skirt and 'touching them inappropriately while they were sleeping'.
The user claimed there was one such 'humiliation room' with 1932 members sharing various 'categories' of porn a 'cousin room', a 'mom room', an 'acquaintance room' and younger and older sister rooms.
Queenarchive1 was eventually suspended from Twitter/X. This was reportedly after many users associated with the forums reported tweets exposing the trend.
Beyond the sharing of real photos, users of the channels reportedly use deepfake technology to create pornographic images of real people with AI.
Two men implicated in the original distribution were sentenced to 40 and 34 years in prison
Earlier this year, US Representative Alexandria Ocasio-Cortez raised awareness of the horrifying use of the technology, telling Rolling Stone she was 'shocked' to see AI-generated 'deepfakes' of herself shared around the internet.
Giorgia Meloni, the Italian prime minister, said this week that deepfake porn videos were a 'form of violence against women' while appearing in court to demand punishment for men who made fake videos of her to be shared online.
Read More
UN issues AI warning and reveal dangers of autonomous weapons and lifelike deepfakes
In many cases, the technology is used to superimpose the face of a real person onto the bodies of pornographic actors or actresses, with potentially very damaging consequences.
Twitter/X user JJorago shared a thread on the resurgence of the 'deepfake porn chats' in South Korea, before women came forward to share their experience of being 'secretly photographed by male relatives'.
'Some were captured changing clothes, while others were photographed sleeping,' reported the Malay Mail.
In other cases, 'profitable' rooms were reported to be using AI to generate nude pictures from just an ordinary, clothed photo of a victim.
The Hankyoreh, a daily newspaper in South Korea, accessed some of these chatrooms and found that AI was being used to 'synthesize a photo of a woman into a nude photo'.
'Upon entering the room, a message immediately appeared in the chat window saying, 'Send me a photo of the woman you like right now.'' a report described.
When the photo was uploaded, an 'illegal composite photo was created' within seconds.
They assessed that as many as 227,000 people were using one room alone.
One victim spoke to the BBC about her experience. She said she received a message from an anonymous sender claiming her pictures and personal information had been leaked.
'Let's discuss,' read the ominous message.
The anonymous victim entered the chatroom to read the full message and received a photo she had taken of herself while still at school.
Another accompanied it - the same photo, but digitally altered and sexually explicit.
More images soon followed. Her face had been edited onto a body 'engaged in a sex act' with AI deepfake technology.
'I was petrified, I felt so alone,' she told the broadcaster.
Hundreds of female celebrities - famous figures including K-Pop singers - have already been named among the victims of deepfake generations shared within these groups.
Won Eun-ji, one of several behind the Team Flame activism group that brought the abuses to light, warned AI bots bring a nefarious new dimension to the problem.
Deepfake content can be made on Telegram for as little as £1.15. These would have been 'unheard of five years ago', she wrote in Hankyoreh recently.
She explained users would find innocent profile shots of their victims and send them in to a group to be altered. Others would pile on, sharing personal details, phone numbers and social media accounts.
Some would send in videos of them masturbating to contact details.
Explicit photos and videos can then be generated from the source material, before being widely circulated with paying members.
'Even as I write this article, messages such as "Anyone want to help me make a deepfake of this 14-year-old I know?" are being posted in the chat room for abuse and humiliation of real-life acquaintances,' Won Eun-ji wrote.
One Telegram channel lists different rooms for different colleges and universities in S Korea
Chatrooms amassing hundreds of thousands have users have again popped up on Telegram
Fear of the damaging nature of this content prompted President Yoon Suk Yeol to issue harsher penalties for sexual deepfakes shared on platforms like Telegram earlier this year.
The ruling People Power Party (PPP) vowed to increase the maximum jail term for sharing sexually explicit fakes from five to seven years - and is pressing Telegram for a hotline to help deal with the scourge.
And late last month, South Korean lawmakers pushed through a bill to criminalise possessing or watching sexually explicit deepfake images and videos, threatening jail time and fines worth up to £17,200. It awaits the President's approval.
The scope of artificial intelligence, coupled with the transmissibility of the internet, stands to completely transform the way we live for the better. But serious questions remain for lawmakers as they try to balance and track possible abuses of this power.
Deepfake technology is among these challenges, with the potential to be used for blackmail, extortion and humiliation of unknowing and unwilling victims. South Korea is taking steps to remedy this problem - but the precedent should unnerve us all.
South Korea
Four years after police concluded a massive bust of depraved content creators in South Korea, deepfake technology brings a sinister new dimension to channels, with hundreds of thousands of people signed up to view digitally altered pictures of women and children.
Despite efforts to crack down on distributors of illegal porn online, users also continue to share photos and videos of strangers, family members and schoolchildren candid and unaware, drugged or sleep, in compromising positions.
Investigations to infiltrate and take down these rooms have uncovered horrifying hazing rituals, with young people even sharing indecent images of their own family members to gain access.
South Korea is battling an online scourge that continues to grow in spite of lengthy sentences and high-profile cases. But with the technology now available world over, the nefarious case of the 'New Nth Room' carries stark implications for us all.
The horror story of similar chatrooms operated 2018-2020 became the focus of a Netflix documentary, Cyber Hell: Exposing An Internet Horror (pictured)
Cho Ju-bin, nicknamed 'Doctor', was only named by police in 2020 after five million people, including local celebrities, signed a petition for him to be identified
One deepfake generating bot tells users: 'Hello sir, I am the magic photo bot. The woman in the picture... well try it and you'll see. Let's start sending pictures of the girl you like right now!' On the right, another post reads: 'Stripped down... invite a friend to process your photos'.
In the original Nth room case, at least 103 people, including minors, were confirmed as victims
In recent weeks, much has been made of the online chatrooms in which users are building communities around the distribution of pornographic pictures.
Investigative journalists are calling the trend the 'New Nth Room', in reference to a similar case that plagued citizens between 2018 and 2020.
Read More
AI is being used to generate deepfake child sexual abuse images of real-life victims, report warns
A virulent string of sex crimes were then exposed by South Korean journalist Kim Wan, who was investigating a student allegedly distributing child pornography through Telegram.
The 'Nth Room case', as it came to be known, saw at least 103 people, including 26 minors, confirmed as victims of a huge underground market for pornographic media sold with cryptocurrency on the messaging platform.
It later emerged that victims were being blackmailed and extorted over the media traded across various channels and groups on Telegram.
With national attention falling on the case, high-profile creators received lengthy jail sentences for their involvement, possibly to deter would-be copycats.
Cho Ju-bin, nicknamed 'Doctor', was only named by police in 2020 after five million people, including local celebrities, signed a petition for him to be identified.
Police said Cho, then 25, threatened at least 74 teens and young women to film explicit videos and used illegal videos of famous faces - including underage celebrities - to bait users into paying to join a Telegram chatroom.
Cho was found guilty that November of running a group which blackmailed girls into sharing sexual videos and sentenced to 40 years in jail.
He apologised 'to those who were hurt by me' and appeared to thank investigators, saying: 'Thank you for putting a brake on the life of a devil that could not be stopped.'
'Doctor' Cho was reported to have been the mastermind behind the operation.
Soon enough, others followed.
Moon Hyung-wook, then 24, was arrested in May 2020 and sentenced to 34 years in prison for 'coercing dozens of women and girls into sharing sexually explicit videos of themselves'.
Moon, known by his online username 'GodGod', was indicted on 12 counts, including violating laws on protecting minors from abuse.
The horror story later became the focus of a Netflix documentary, Cyber Hell: Exposing An Internet Horror.
Cho Ju-bin, known as the 'Doctor', was sentenced to 40 years for leading the online sex ring
Moon Hyung-wook, known by his online username 'GodGod', was sentenced to 34 years
Despite the public arrests and strong sentences, South Korean outlets suggest we may be witnessing a revival of a similar trend - now assisted by use of AI.
Koreaboo reported recently that some of the groups require new members to pass a kind of hazing test by sharing illicit photos of their mothers, sisters, and other women in their lives - 'and provide the girl's personal details'.
Read More
Giorgia Meloni tells court deepfake porn videos of her are 'a form of violence against women'
'Some of these rooms revolve around 'humiliating' one's family members by taking photos of videos of oneself inappropriately touching one's sister or mother,' the outlet reported.
Horrifically, the images reportedly extend to the distribution of pictures of minors.
Some women are asleep or drugged, according to the Malay Mail. Some users are pictured 'acting inappropriately' with the victims, including mothers, siblings or relatives'.
'Most victims were middle and high school students,' Koreaboo claimed.
'When one participant hesitated to abuse an elementary school student born in 2013, others criticised him, saying: 'Her age is double digits. What's the problem?''
The outlet, which has been central in helping to identify and expose the groups online, reported that 'various sources and bokep chindo victims' have uncovered rooms with a total of more than 212,000 participants.
Twitter/X user Queenarchive1 brought the issue to light with a series of posts of Telegram rooms.
A screenshot of one of the rooms reportedly showed one of the users lifting their younger sister's skirt and 'touching them inappropriately while they were sleeping'.
The user claimed there was one such 'humiliation room' with 1932 members sharing various 'categories' of porn a 'cousin room', a 'mom room', an 'acquaintance room' and younger and older sister rooms.
Queenarchive1 was eventually suspended from Twitter/X. This was reportedly after many users associated with the forums reported tweets exposing the trend.
Beyond the sharing of real photos, users of the channels reportedly use deepfake technology to create pornographic images of real people with AI.
Two men implicated in the original distribution were sentenced to 40 and 34 years in prison
Earlier this year, US Representative Alexandria Ocasio-Cortez raised awareness of the horrifying use of the technology, telling Rolling Stone she was 'shocked' to see AI-generated 'deepfakes' of herself shared around the internet.
Giorgia Meloni, the Italian prime minister, said this week that deepfake porn videos were a 'form of violence against women' while appearing in court to demand punishment for men who made fake videos of her to be shared online.
Read More
UN issues AI warning and reveal dangers of autonomous weapons and lifelike deepfakes
In many cases, the technology is used to superimpose the face of a real person onto the bodies of pornographic actors or actresses, with potentially very damaging consequences.
Twitter/X user JJorago shared a thread on the resurgence of the 'deepfake porn chats' in South Korea, before women came forward to share their experience of being 'secretly photographed by male relatives'.
'Some were captured changing clothes, while others were photographed sleeping,' reported the Malay Mail.
In other cases, 'profitable' rooms were reported to be using AI to generate nude pictures from just an ordinary, clothed photo of a victim.
The Hankyoreh, a daily newspaper in South Korea, accessed some of these chatrooms and found that AI was being used to 'synthesize a photo of a woman into a nude photo'.
'Upon entering the room, a message immediately appeared in the chat window saying, 'Send me a photo of the woman you like right now.'' a report described.
When the photo was uploaded, an 'illegal composite photo was created' within seconds.
They assessed that as many as 227,000 people were using one room alone.
One victim spoke to the BBC about her experience. She said she received a message from an anonymous sender claiming her pictures and personal information had been leaked.
'Let's discuss,' read the ominous message.
The anonymous victim entered the chatroom to read the full message and received a photo she had taken of herself while still at school.
Another accompanied it - the same photo, but digitally altered and sexually explicit.
More images soon followed. Her face had been edited onto a body 'engaged in a sex act' with AI deepfake technology.
'I was petrified, I felt so alone,' she told the broadcaster.
Hundreds of female celebrities - famous figures including K-Pop singers - have already been named among the victims of deepfake generations shared within these groups.
Won Eun-ji, one of several behind the Team Flame activism group that brought the abuses to light, warned AI bots bring a nefarious new dimension to the problem.
Deepfake content can be made on Telegram for as little as £1.15. These would have been 'unheard of five years ago', she wrote in Hankyoreh recently.
She explained users would find innocent profile shots of their victims and send them in to a group to be altered. Others would pile on, sharing personal details, phone numbers and social media accounts.
Some would send in videos of them masturbating to contact details.
Explicit photos and videos can then be generated from the source material, before being widely circulated with paying members.
'Even as I write this article, messages such as "Anyone want to help me make a deepfake of this 14-year-old I know?" are being posted in the chat room for abuse and humiliation of real-life acquaintances,' Won Eun-ji wrote.
One Telegram channel lists different rooms for different colleges and universities in S Korea
Chatrooms amassing hundreds of thousands have users have again popped up on Telegram
Fear of the damaging nature of this content prompted President Yoon Suk Yeol to issue harsher penalties for sexual deepfakes shared on platforms like Telegram earlier this year.
The ruling People Power Party (PPP) vowed to increase the maximum jail term for sharing sexually explicit fakes from five to seven years - and is pressing Telegram for a hotline to help deal with the scourge.
And late last month, South Korean lawmakers pushed through a bill to criminalise possessing or watching sexually explicit deepfake images and videos, threatening jail time and fines worth up to £17,200. It awaits the President's approval.
The scope of artificial intelligence, coupled with the transmissibility of the internet, stands to completely transform the way we live for the better. But serious questions remain for lawmakers as they try to balance and track possible abuses of this power.
Deepfake technology is among these challenges, with the potential to be used for blackmail, extortion and humiliation of unknowing and unwilling victims. South Korea is taking steps to remedy this problem - but the precedent should unnerve us all.
South Korea