The XZ Utils backdoor, discovered last week, and the Heartbleed security vulnerability ten years ago, share the same ultimate root cause. Both of them, and in fact all critical infrastructure open source projects, should be fixed with the same solution: ensure baseline funding for proper open source maintenance.

Open source software is the foundation of much of our digital infrastructure. From web servers to encryption libraries to operating systems, open source code powers systems that millions rely on daily. The open source model has proven tremendously successful at producing innovative, reliable, and widely used software.

However, the Heartbleed vulnerability in OpenSSL and the recent backdoor discovered in the XZ Utils compression library have highlighted potential weaknesses in how open source software is funded, developed, and maintained. These incidents showed that even very widely used open source projects can have serious, undiscovered bugs due to lack of resources.

Learnings from Heartbleed

Today, April 7th, 2024, marks the 10-year anniversary since CVE-2014-0160 was published. This security vulnerability known as “Heartbleed” was a flaw in the OpenSSL cryptography software, the most popular option to implement Transport Layer Security (TLS). In more layman’s terms, if you type https:// in your browser address bar, chances are high that you are interacting with OpenSSL.

The fallout from Heartbleed was immense, prompting widespread panic among developers, businesses, and users alike. About one-fifth of all web servers in the world at the time were believed to be vulnerable to the attack, allowing theft of the servers’ private keys and users’ session cookies and passwords.

The software bug existed in OpenSSL’s codebase for over two years before being discovered. While code reviews were in place, the bug wasn’t spotted and went into OpenSSL’s source code repository on New Year’s Eve, December 31st, 2011. At the time, the OpenSSL project was maintained by a small 4-person team with limited funding and basically working as volunteers, driven by just the importance of their mission.

This was the ultimate root cause – a piece of software that had started as a hobby project (just like Linux) grew over time and became part of the Internet infrastructure, but there was no mechanism to ensure resources would grow to be able to maintain it well long-term.

In April 2014, the Linux Foundation Executive Director Jim Zemlin seized the opportunity to get visibility and managed to get Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Qualcomm, Rackspace, and VMware to all pledge to commit at least $100,000 a year for at least three years to the Core Infrastructure Initiative. The initiative continued for many years and eventually transformed into the Open Source Security Foundation. Also due to Heartbleed, the European Commission launched the EU-Free and Open Source Software Auditing project and spent at least a million euros on auditing OpenSSL, the Apache Server, KeePass, and other security-critical open source software.

This relatively modest funding, along with code audits and process improvements, allowed OpenSSL to become more secure and sustainable. Today the OpenSSL project is thriving: it is FIPS 140-2 certified and has a healthy base of both financial and code contributors.

Learnings from the XZ / liblzma library backdoor

While there are surely still more details to uncover in the coming weeks, when the news broke about the XZ compression software backdoor (CVE-2024-3094), it was immediately clear that it happened because XZ had become hugely popular and widely used but was still maintained by one single overworked person as a spare time project. A well-resourced malicious actor was able to manipulate and pressure the maintainer to give them commit access, and thus the software supply chain was compromised. We should not blame the original maintainer, but rather everyone else for not realizing how widely used XZ was, yet going by with very little support and resources.

A huge number of applications depend on XZ. Right now the priority should be to offer help to maintain it properly, both upstream and at various downstreams, such as in Linux distributions, so the whole software supply chain is secured. It does not require a massive effort – just having a couple more maintainers to share the maintenance and review work should go a long way.

Would we be better off with closed-source software?

In both cases, the vulnerabilities were fixed quickly because the world had access to the source code of the affected software. This is a major advantage of open source software: it allows anyone to inspect the code and find potential vulnerabilities, and submit fixes to them.

In the case of Heartbleed, Google’s security team reported it to OpenSSL first, but the Finnish national NCSC-FI has records of local cybersecurity company Codenomicon reporting it independently. In the case of XZ, a Microsoft employee and PostgreSQL developer Andres Freund found the backdoor while doing performance regression testing in a Debian Linux development version. It was a huge fluke of luck that the XZ backdoor didn’t go in any actual Linux distribution releases. Next time we might not be as lucky, so more reviews, testing, and validation are needed. It will need resources, but at least public review is possible – thanks to this infrastructure-level software being open source.

Public scrutiny, testing, and validation are not possible for closed-source software. In fact, if closed-source code gets backdoored, it will go unnoticed for a much longer time. For example, the 2020 U.S. government data breach was possible due to multiple backdoors and flaws that went undetected for a long time in closed-source software from SolarWinds, Microsoft, VMware, and Zerologon. In theory, companies always have money (unless they are bankrupt), but in practice, the pressure to channel that money into software review and testing varies wildly, and working without exposure to public scrutiny often incentivizes companies to skimp on security to maximize profits.

Thus, I firmly believe in open source software having a better overall security posture as long as there are reasonable resources. And if the source code is public, anybody can audit how active the maintenance is and thus also the fact if maintenance is funded itself is a public and auditable property of open source.

Pledge for funding and participation

Both Heartbleed and the XZ backdoor incident underscore the critical role that open source software plays in powering the digital infrastructure of today’s world. Such important and widely used projects shouldn’t be struggling to get by. It’s time for companies to step up and provide reasonable funding to the projects they depend on.

You don’t need billions to meaningfully improve open source security – the OpenSSL example shows that even modest funding increases can have an outsized impact. A tiny slice of the corporate IT budget pie could go a long way. Additionally, some of the government defense spending should be funneled into key open source software projects that our society relies on.

The incidents of Heartbleed and the XZ backdoor serve as sobering reminders of the vulnerabilities that may exist within our open source infrastructure today. However, they also present an opportunity for positive change. By investing in the security and maintenance of open source projects through moderate funding and support, we can enhance the resilience of our digital infrastructure and ensure a safer and more secure internet for all.