Inside the Race to Fix a Potentially Disastrous Software Flaw

(Bloomberg) -- At 2:51 p.m. on Nov. 24, members of an open-source software project received an alarming email. The contents threatened to undermine years of programming by a small group of volunteers and unleash massive cyberattacks across the globe.

“I want to report a security bug,” wrote Chen Zhaojun, an employee on Alibaba Group Holding Ltd.’s cloud-security team, adding “the vulnerability has a major impact.”

The message went on to describe how a hacker could take advantage of Log4j, a widely used software tool, to achieve what’s known as remote code execution, a hackers’ dream because they can remotely take over a computer. 

The message ultimately set off a global race to update critical computer systems, with senior U.S. cybersecurity officials describing the discovery as a “significant threat.” Left unfixed, the software could give attackers unfettered access to untold millions of computer systems.

But behind the scenes, a small cadre of unpaid programmers went to work to patch the faulty software.

Log4j is a piece of software that developers can put into applications to monitor, or "log," anything from mundane operations to critical alerts. Those detailed logs can help programmers debug software. According to security researcher Marcus Hutchins, Log4j is used by millions of applications.

It is open-source software that is maintained by a group of volunteer programmers as part of the nonprofit Apache Software Foundation, one of dozens of open-source projects that have become a crucial component of global commerce and that are mostly maintained by unpaid volunteers.

Interviews and documents obtained by Bloomberg News reveal, for the first time, their minute-by-minute efforts to shore up a software flaw that has the potential to be one of the most damaging cybersecurity incidents in recent memory. 

“Some security issues you get are sort of red herrings,” said Gary Gregory, who has worked on the Apache Software Foundation team that maintains Log4j for nearly a decade. “But this one was, ‘Oh crap.’ In this case, some of us were surprised, not that there was a security issue, but just how bad it was.”  Gregory, who has a full time job as a principal software engineer at Rocket Software, said he works for free on open source projects because he enjoys it. “I love writing software. It's my passion."

After receiving the email from Chen, Apache’s volunteer programmers began working to fix the vulnerability before the rest of the world knew there was a problem.

But on Dec. 8, the team received another email from Alibaba’s Chen, notifying them that someone had just revealed the details of the vulnerability on a Chinese blogging platform for the entire internet to see. “Some WeChat security chat groups are already discussing the details of the vulnerability, and some security researchers already have the vulnerability,” Chen wrote. “We promise to keep it secret until your official release version comes out. Please hurry up.”

Chen didn’t immediately respond to an email seeking comment.  The person who published the details of the flaw, who uses a pseudonym, didn’t respond to a request for comment.

By then, hackers had already started exploiting the flaw, according to a tweet by CloudFlare Chief Executive Officer Matthew Prince. Some 20 hours later, Apache’s team working on Log4j published a “patch” to fix the problem. That’s when hackers began “mass exploitation” of the flaw, according to Prince.

In the frantic time since the flaw was publicly disclosed, researchers have concluded that the vulnerability had existed in Log4j since September 2013, apparently unknown to its vast universe of users.

However, the impact of the Log4j flaw remains unknown. Because the software exists in so many products and services, it may be months -- or even years -- before every version of it receives the update, according to security experts. So far, no major hacks have been tied to the vulnerability.

Security researchers have found applications by Apple, Twitter and dozens of other companies using Log4j. There is no indication any have suffered a security breach as a result.

The disclosure of the vulnerability was personally gutting for Apache’s volunteer contributors, some of whom published thousands of “commits,” or changes, to improve the code over the years. One former Log4j developer, Christian Grobmeier, now a vice president at the Apache Software Foundation, was taken aback on the morning of Friday, Dec. 10, when he opened his inbox and first learned the news amid a flurry of messages from Log4j volunteers.

“I was thinking, ‘Oh my God. This is my project. And then I was thinking, Apple is involved. Twitter is impacted. Everything,” he said. “And then, I was just realizing, how many people were using using this software. This is basically half of the world, maybe even more. This is just crazy.”

He described the conversations among the Log4j group as dispassionate and earnest. “I know these people -- they all have families and things they have to do. But they put everything aside and just sat down for the whole weekend and worked on that,” he said.

The vulnerability discovered in Log4j highlights how much modern software relies on open source projects maintained by unpaid volunteers, and what happens when a major security vulnerability is discovered within one. 

“The scenarios are endless,” said Mark Curphey, founder of the Open Web Application Security Project, a nonprofit focused on open-source cybersecurity projects and co-founder and chief technology officer of cybersecurity startup Open Raven. “Reusable code means reusable vulnerabilities and software supply chains feed the world’s technology consumers. There lies a perfect storm.”

Lou Steinberg, founder and managing partner at CTM Insights and former chief technology officer of TD Ameritrade, said it was likely millions of servers were at risk. “We are in a race against attackers to patch before they can exploit.” 

Since the flaw was disclosed, some anger has been directed at Apache’s developers.  And there were warning signs that that Log4j may be vulnerable, including at a presentation at the Black Hat cybersecurity conference in 2016. There, researchers identified a method to exploit a broad class of software that included Log4j, according to Daniel Stenberg, who created one of the world’s most-used pieces of open-source software, called Curl, that is used for transferring data between applications.

“Why wasn't it fixed then? I really don't know,” Stenberg said. “It seems the Logj4 authors really didn't understand that they had a ticking bomb in their code even after that was highlighted. Clearly the Log4j project needed an outsider to poke them in the eye and really make the aware of the problem. How do you force that to happen? Not easy.”

©2021 Bloomberg L.P.