I asked benny Vasquez, chair of the AlmaLinux OS Foundation, how she would explain the recent Red Hat Enterprise Linux source code controversy to somebody at a family barbecue—somebody who, in other words, might not have followed the latest tech news quite so closely.
“Most of my family barbecues are going to be explaining that Linux is an operating system,” Vasquez said. “Then explaining what an operating system is.”
It is indeed tricky to explain all the pieces—Red Hat, Red Hat Enterprise Linux, CentOS, CentOS Stream, Fedora, RHEL, Alma, Rocky, upstreams, downstreams, source code, and the GPL—to anyone who isn’t familiar with Red Hat’s quirky history, and how it progressed to the wide but disparate ecosystem it has today. And, yes, Linux in general. But Vasquez was game to play out my thought experiment.
“The changes that have recently been made are best summed up as: Red Hat has historically made it easy for what they view as competitors to exist,” she said. “And the changes they’ve made, they think, make it less easy for competitors to exist. From a high-level perspective, for people who don’t understand ‘build pipelines,’ that’s how I would want to explain it.”
“We can fix this now. We don’t have to wait.”
AlmaLinux OS, until recently, aimed to be a “1:1,” or “bug for bug” replication of Red Hat Enterprise Linux (RHEL). When RHEL announced that its source code would only be available in CentOS Stream, the “rolling preview” of RHEL, it made creating a 1:1 rebuild of RHEL far more tricky. Rocky Linux, founded by one of the original CentOS’s founders, has said it intends to keep providing bug-for-bug rebuilds through some elaborate means.
AlmaLinux, after waiting out the initial confusion and surveying its customers and supporters, is going a different route. AlmaLinux will be binary-compatible (or ABI-compatible), meaning applications that run on RHEL will run on AlmaLinux. Freed from complete parity with RHEL releases, however, means that AlmaLinux can:
- Accept bug fixes outside RHEL’s release cycle
- Include comments in patches that point to sources and authors
- Decide its own priorities
- Continue contributing upstream to CentOS Stream, Fedora, and Linux as a whole
“Now we can do stuff!” Vasquez said. “That’s exactly how it’s been feeling for us. We’ve used that one-to-one compatibility as our North Star, so every decision we’ve made about what we’re doing has been, yes or no, based on one-to-one compatibility. This opens up so many doors.”
One of those doors, it seems, is security patches undertaken quite differently from RHEL. Jonathan Wright, infrastructure team lead at AlmaLinux and a Fedora package maintainer, recently posted about his experience submitting a pull request, based on an existing CVE (vulnerability), to CentOS Stream. Michal Ruprich, senior software engineer at Red Hat, replied in GitLab that RHEL didn’t plan to address it, but “we will keep it open for evaluation based on customer feedback.” On further querying by Wright, Ruprich replied that vulnerabilities with low or moderate severity are addressed “on demand when customer or other business requirement exist to do so.”
There was more context, of course, but the moment served as a kind of proof of concept for the new AlmaLinux. “It is an example of what we wanted to be able to do, what we were hoping this would be… we can fix this now. We don’t have to wait.”
Red Hat responds
Red Hat made a point of calling out “those who want to repackage (RHEL) for their own profit” in a follow-up blog post, soon after its initial announcement. Citing “large or very large IT organizations” that use RHEL rebuilds without supporting Red Hat itself, the company said it did not “find value in an RHEL rebuild.”
I asked Red Hat if it had anything further to say about rebuilds in the wake of AlmaLinux OS’s shift. I also asked about the “customer feedback” response to the security patch. Mike McGrath, vice president of core platforms at Red Hat, responded with a statement. McGrath wrote that after hearing the feedback after the source changes, he wanted to “reaffirm our commitment to open source.” He said that Red Hat “honor(s) and in most cases exceed(s) all of our license obligations,” that source code for all Red Hat’s products is made available, and that Red Hat customers still have source access to RHEL. McGrath also pointed to Red Hat Universal Base Image, the no-cost Individual Developer subscription, and Teams subscriptions as fulfilling open source goals.
“With all of these options, we just don’t see any reason to provide source code in yet another location, scrubbed of our trademarked material, for the sole purpose of creating ‘bug for bug’ compatible clones,” McGrath wrote. “We would rather work together in CentOS Stream instead, where improvements are possible. At least one of the formerly downstream communities has already made the decision to work from CentOS Stream sources, and we applaud this shift and are eager to collaborate with them, even if we ultimately compete in a business sense. Differentiated competition is a sign of a healthy ecosystem.”
What, then, of the recent rejection of just such an offer to help improve CentOS Stream through a CVE fix? McGrath addressed that specifically.
“Building RHEL is incredibly complex and resource intensive—there’s tens of thousands of moving parts, and all of this is on display in CentOS Stream,” McGrath wrote. “With an emphasis on production stability, we aren’t able to immediately take every patch or merge request—this is the crux of the recent issue surrounding a CVE patch from an AlmaLinux contributor. At the time of submission, the CVE didn’t have a public severity assessment done and Red Hat hadn’t finished its independent assessment either. We didn’t close the merge request and continue to evaluate it for future inclusion.”
“It’s also already been accepted to Fedora; this means that it will, eventually, be included in RHEL,” McGrath wrote. “When it comes to enterprise Linux, being deliberate, predictable and thorough is key—that’s what this process shows, even in the supporting upstream community.”