Gov't, public and parents on alert as deepfake sex crimes rise in Korea

A worrying surge in the alleged creation and distribution of deepfake pornography through Telegram chat rooms has exposed a worrying trend regarding teenage involvement in the crime.

Activists hold masks and shout slogans during a protest against deepfake porn in Gangnam District, southern Seoul, on Friday. [AFP/YONHAP]

A surge in the alleged creation and distribution of deepfake pornography through Telegram chat rooms has exposed a worrying trend regarding teenage involvement in the crime, prompting the Korean government and law enforcement to take urgent action.

The government on Friday announced plans to establish new regulations that will penalize not only the possession and purchase of deepfake videos but also the viewing of material.

A unified reporting channel for such content will also be launched to streamline current reporting methods across ministries.

The government also agreed with the conservative People Power Party on Thursday to establish a "hotline" with Telegram and adopt longer prison terms for those who create and distribute sexually explicit content, from five years to seven years.

An intensive crackdown on deepfake sex crimes began Wednesday, following President Yoon Suk Yeol’s order to “eradicate” the issue.

Chat rooms gain attention

These measures come as numerous chat rooms were linked to hundreds of schools — from universities to elementary schools — as well as military units. The rooms were accused of producing and distributing deepfake pornographic content on Telegram, with some of the chat rooms reportedly containing around 1,000 participants.

Participants in the chat rooms allegedly used photos of acquaintances, mostly women, taken from social media platforms like Instagram, to create and distribute sexually explicit images and videos using deepfake technology.

Police launched an investigation on Aug. 20 into one of these chat rooms targeting female students at Inha University.

Active since 2020 with about 1,200 participants, this chat room was used to share digitally altered images of female students, including personal information such as phone numbers and addresses.

On Friday, one of the deepfake crime suspects, recently arrested on charges of creating and distributing 279 deepfake porn videos on Telegram between May and July, was transferred to the prosecution for possible indictment, according to the Seoul Metropolitan Police Agency.

The man in his 20s ran a Telegram channel called "Acquaintance Humiliation Room," which allegedly victimized as many as 246 people.

As the scale of these crimes expands and fears mount over potential victimization, a map displaying schools suspected of being targeted by deepfake pornography gained widespread attention on social media. Created by a middle school student, the map marked over 500 locations and garnered more than 3 million visits within a day after it was posted.

A screen capture of a map by which individuals can look up which schools have been affected and their locations. The map was allegedly created by a middle school student after he heard that his classmates also fell victim to recent deepfake crimes. [SCREEN CAPTURE]

Teenage involvement An alarming aspect of this snowballing issue is that teenagers are both the primary victims and main perpetrators of the crimes.

The Korean Teachers and Education Workers Union identified approximately 520 cases of indirect and direct damage from deepfake sexual exploitation among students and teachers nationwide between Aug. 27 and 28 alone.

This data is not unexpected, as recent police findings indicate that minors made up six out of every 10 victims of deepfake sex crimes between 2021 and last year.

According to data submitted on Friday by the Korean National Police Agency to liberal Democratic Party lawmaker Yang Bu-nam, 59.8 percent of the 527 people who fell victim to deepfake pornography between 2021 and 2023 were teenagers.

The number of teen victims more than tripled from 2021 to 2023.

The percentage of teenagers charged with creating illicit deepfake content also rose from 65.4 percent in 2021 to 75.8 percent in 2023, with the figure at 73.6 percent for the period from January to July this year.

Won Eun-ji, an activist from 56flame, appeared on CBS Kim Hyun-jung’s News Show on Aug. 28 and said that teenagers played a significant role in promoting the recently discovered deepfake chat rooms.

Activist group 56flame first exposed the existence of the notorious “Nth room” in 2019, whose participants illegally produced and distributed pornographic material containing images of females.

Deepfake porn has become a business in the chat rooms, with many teenagers paying money to gain access to content.

"Teenagers advertise these chat rooms or share digitally altered photos and videos in other chat rooms to earn credits,” Won explained. The credits, which can be used as currency to buy or commission deepfake pornographic content from the Telegram chat rooms, are worth approximately 2,650 won ($1.98) each.

“Making a pornographic video requires five credits, so the deepfake pornography video is about 13,260 won,” Won said.

Won also mentioned that individuals can earn credits by inviting others to join these chat rooms, a tactic that particularly appeals to financially constrained teens. This results in the chat rooms being spread widely among peers.

If someone pays to have pornography made, they are also granted access to chat rooms where deepfake images of numerous anonymous women are shared.

According to Won, the victims of deepfake sexual exploitation are mainly women, which is consistent with a study by the international cybersecurity firm Security Hero. The study, conducted between July and August last year, examined the top 10 deepfake porn websites and 85 channels on YouTube and Dailymotion.

It found that 53 percent of the individuals shown in deepfake pornographic content were from Korea, with 99 percent of the content featuring women.

Members of the Korean Teachers and Education Workers Union hold a press conference at the Government Complex in Jongno District, central Seoul, on Thursday to announce the results of an investigation into deepfake-related sex crimes in schools. [YONHAP]

Barriers to investigation

While investigations are ongoing to identify victims, track down perpetrators and assess the full extent of the damage, efforts are being hampered by Telegram's end-to-end encryption.

The app's strong privacy protection prevents anyone, including law enforcement agencies, from accessing the encrypted data stored on servers or transmitted over networks, as well as users’ information, even with a court order, according to authorities.

Telegram's history of non-cooperation with investigations further complicates the situation.

“Telegram does not cooperate with investigations according to their own principle, making it impossible for prosecutors to carry out investigations even when they have a court order,” said a prosecutor.

The app has been controversial since its launch in August 2013, with serious crimes related to sexual exploitation, drug trafficking and money laundering allegedly occurring on the app. Yet, Telegram has never complied with investigative authorities.

Meanwhile, the number of active Telegram users in Korea has grown from 2.86 million in December 2021 to 3.16 million in July 2023, according to Mobile Index, a domestic app analysis service.

Public looks for answers

A hashtag demanding the exposure of deepfake perpetrators' online profiles is currently trending across the country, with over 167,000 posts on online platform X as of Sunday. Many of these posts are urging the nation’s education offices to expel teenage offenders.

Additionally, the hashtag #Protect_WomenAndGirls has gained significant traction, surpassing 41,580 posts as of Sunday and trending in Australia.

As investigations are expected to progress slowly, some netizens began sharing lists of suspected perpetrators on social media platforms, leading to wrongful targeting and some individuals receiving hate messages and phone calls after being mistakenly identified as suspects.

"I was wrongly accused of being a deepfake perpetrator, but I'm just a mother of two children," said one netizen on X, who received threatening messages after her Instagram account was wrongly listed as being connected to deepfake activities.

Self-preventive measures are also emerging online as people become aware of the massive scale of the case.

Many parents online are sharing lists of schools that might be targeted in their areas and are advising each other to monitor their children's phones and to closely supervise the use of Telegram.

"You must check your children's phones. Don’t assume your kids are not at risk,” said a concerned mother on an online community for mothers.

Some individuals are already taking steps to protect their privacy by deleting photos of themselves or making their online profiles private. A mother of two residing in Gumi, North Gyeongsang, recently removed her photos on social media, which featured her children.

"Although my children are both boys, I deleted all of their photos on social media," she said.

People are also seeking help from digital undertakers — companies that use big data programs to find and erase unwanted personal information from the web.

Kim Ho-jin, CEO of SantaCruise Company, which specializes in removing personal data, reported a surge in deepfake-related inquiries during an interview with local media YTN on Wednesday.

"I usually receive one or two inquiries a week, but recently, I got over 30 calls related to deepfakes in just a week," Kim said. He noted that initially, most of these requests came from parents of teenage victims.

However, recently, he has seen a surge in calls from some perpetrators as well, who are now seeking to have their information that was exposed online removed out of fearing severe consequences.

Ko Kwang-pyo of the Women and Juvenile Affairs Division at Daejeon Seobu Police Station speaks at an education session on measures to prevent deepfake-related sex crimes for students at a high school in Daejeon on Friday. [NEWS1]

Government response

In response to the growing crisis, law enforcement is ramping up efforts. Each city and district's digital sex crime investigation team is employing programs to detect deepfake content and examining Telegram bots that automate content creation. The investigation will also extend internationally.

School police officers are also conducting educational classes on these crimes and will gather intelligence, as many teenagers are currently known to be involved in such crimes.

In Seoul, the Seoul Metropolitan Police Agency, in collaboration with the Seoul Metropolitan Office of Education, has activated the "School Bell" alert system across 1,374 schools, reaching 780,000 parents.

The "School Bell" system sends alerts through an application or text messages to students, teachers and parents when a new teenager-related crime emerges.

The seriousness of deepfake content distribution also prompted Korean platform operators to boost their surveillance. Naver, the country’s internet conglomerate and search engine, said on Wednesday that it is operating the Clova GreenEye 2.0 system on its platform to detect any sexually explicit content uploaded to the portal. The system will automatically delete content deemed inappropriate, Naver said.

KakaoTalk, the country's No.1 messaging app, established a channel accepting reports of any digital sex crime using deepfake technology.

Protecting yourself

If an individual sees their photos being illegally used in deepfake pornography, police and IT experts advise that they should immediately set their social media accounts to private and delete any personal information posted online.

Victims should then capture evidence of the content, verify if was made using their original photos and report it to the police or the Korea Communications Standards Commission.

BY WOO JI-WON [woo.jiwon@joongang.co.kr]