Experts blast government's shoddy response to administrative portal crash

The government faces a massive backlash over what critics call its belated and deficient response to a network glitch that paralyzed the nationwide administrative network last week.

Public workers check whether the administrative network is running smoothly after recovery at Dongjak District office in southern Seoul on Sunday. [YONHAP]

The government faces a massive backlash over what critics call its belated and deficient response to a network glitch that paralyzed the nationwide administrative network last week.

The IT industry and experts criticize the government's failure to precisely identify the cause even after taking 50 hours to develop solutions.

Officials revealed Wednesday that they had no disaster manual to resolve digital crises at the central government level.

Minister of Interior and Safety Lee Sang-min was in the United States to boost Korea’s profile as a leading country in digitalized public administration and governance when the country suffered an electronic governance outage.

As the situation unfolded ominously, he returned a day earlier than scheduled.

The glitch in the Saeol administrative network, used by civil servants to access government documents, occurred Friday morning. Later that day, the glitch even interrupted the network of Government24, the central government’s one-stop civil service portal.

Although the ministry attempted to normalize the service when the malfunction was detected, it failed. Both services remained down until the end of business hours on Friday.

The government attempted to restore the Government24 and Saeol network services over Saturday and Sunday.

By early dawn on Saturday, a recovery team of public officials and private sector experts replaced the network equipment and stabilized the connection, according to the ministry.

The operating servers and supporting equipment are at the National Information Resources Service in Daejeon. The Interior Ministry manages and oversees the information service.

“We tested whether the replacement is stable enough to handle overflowing traffic and interoperable with the existing network,” said Suh Bo-ram, director-general of Korea's digital government bureau under the Interior Ministry.

On Sunday, Vice Interior Minister Ko Ki-dong apologized for the outage, adding that the government will retroactively handle civil service requests that were not addressed promptly on Friday.

“The ministry could confirm that the Saeol network is running smoothly based on the two-day inspection results," Ko said. "All local administrative services are now back on track. The ministry will continue monitoring to ensure additional mishaps or inconveniences do not happen again.”

Interior Minister Lee Sang-min chairs a meeting on Monday at the government complex in central Seoul to discuss the response to the nationwide network crash last Friday. [YONHAP]

The ministry also revealed that the cause of the crash was an error in the authorization system’s network device, the so-called Government Public Key Infrastructure (GPKI). Public officials must go through GPKI to process administrative and civil requests at district, city and provincial offices.

The Government24 service, which uses the GPKI, became completely paralyzed.

Private sector professionals found the error at the L4 switch, which distributes online traffic to various servers. The switch is connected to the GPKI.

The ministry said an error at the L4 switch disrupted the user authorization process, preventing users from accessing the administrative network.

Some say Thursday's update to the information service’s security program triggered the network failure. As per industry practice, updates are usually carried out on weekends, when traffic volumes are lower.

The ministry has not yet provided a detailed explanation of how and why the L4 switch malfunctioned.

IT professionals accuse the government of failing to identify the root causes of the crash in a proper and timely manner.

“The ministry could confirm the specific cause of the glitch after a more thorough investigation,” Seo said. “We prioritized bringing the service back online. Fixing the problem came before identifying the cause.”

National Information Resources Service in Daejeon [JOONGANG PHOTO]

The Framework Act on the Management of Disaster and Safety, which defines the 41 situations where a crisis manual must be in place, does not include the crash of an administrative network system, according to the Interior Ministry.

The National Information Resources Service relied on internal guidelines to restore the network system. However, as the guidelines only entail recovery solutions at the working level, they provided no instructions on coordinating with affiliated governmental agencies and offices.

Interior Minister Lee Sang-min, third from the left, participates in the 2023 Digital Nations Ministerial Meeting in Portugal on Nov. 15. The meeting was intended to facilitate the digitalization of public administration and governance. He later flew to the United States to discuss international cooperation in public administration with U.S. counterparts. [YONHAP]

The ministry also disappointed people with unsatisfactory post-crisis management. Though the crisis could have affected the entire public, the authorities did not take proactive measures to inform people.

There were no public alerts, and people were left unnotified for several hours Friday until a news release came out at noon Friday.

This stands in striking contrast to the KakaoTalk crash last October, where the Ministry of Science and ICT sent out an emergency alert three times.

“This crash does not fulfill the condition of an emergency. That’s why the government did not send out an emergency alert,” the Interior Ministry official said. “The comparison with the KakaoTalk crash is improper because the server was burned and inflicted a huge loss.”

The government developed a hindsight solution to process civil service requests in handwriting.

The private sector IT Industry says that governmental responses are hardly acceptable.

From their view, there is a double standard where the government applies stricter rules and mechanisms on private firms while being forgiving regarding its own mistakes.

“If private companies such as Naver and Kakao took measures in the way that the government did, the governmental authorities would not tolerate it,” an IT expert said.

According to another expert from a portal company, the government usually chases them to submit a report and restore the service as fast as they can, even when the error only lasts 30 minutes. The government used to justify the pressure by citing the massive number of users.

“I am highly doubtful whether the government deserves to hustle private portal and IT companies and platform providers,” he said.

“An effort to identify the cause and restore the service can delay the government’s statement preparation. Even though the government’s position has been turned from criticizer to one who bears responsibility, its responses are far different [from its standard levied on private companies],” said Kim Seung-joo, a cyber-security professor at Korea University.

“Regarding IT problems, high-ranking officials have never taken their full responsibility. This pattern repeats the problem. Just cutting the ties with the outsourced companies or putting the blame on working-level personnel would be a most undesirable closure,” Kim added.

BY NA-UN CHAE, AN DAE-HUN, LEE SOO-JUNG [lee.soojung1@joongang.co.kr]