These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.
I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.
I’ve been on the CentOS Project Governing Board since its creation. I also was part of the consensus decision that we recently announced about shifting the project’s focus. I’ve cared about this space for a long time, for my 19 years at Red Hat and prior to that. I was involved in the Fedora Project since the earliest days, leading the documentation project and sitting on the then-Fedora Board, among other roles. I led the team at Red Hat that brought the CentOS Project in closer to Red Hat in 2013/2014, and as a result of that work I earned a seat on the CentOS Governing Board, where I was the Red Hat Liaison and Board Secretary until Spring 2020.
Let’s go back to 2003 where Red Hat saw the opportunity to make a fundamental change to become an enterprise software company with an open source development methodology.
To do so Red Hat made a hard decision and in 2003 split Red Hat Linux into Red Hat Enterprise Linux (RHEL) and Fedora Linux. RHEL was the occasional snapshot of Fedora Linux that was a product—slowed, stabilized, and paced for production. Fedora Linux and the Project around it were the open source community for innovating—speedier, prone to change, and paced for exploration. This solved the problem of trying to hold to two, incompatible core values (fast/slow) in a single project. After that, each distribution flourished within its intended audiences.
But that split left two important gaps. On the project/community side, people still wanted an OS that strived to be slower-moving, stable-enough, and free of cost—an availability gap. On the product/customer side, there was an openness gap—RHEL users (and consequently all rebuild users) couldn’t contribute easily to RHEL. The rebuilds arose and addressed the availability gap, but they were closed to contributions to the core Linux distro itself.
In 2012, Red Hat’s move toward offering products beyond the operating system resulted in a need for an easy-to-access platform for open source development of the upstream projects—such as Gluster, oVirt, and RDO—that these products are derived from. At that time, the pace of innovation in Fedora made it not an easy platform to work with; for example, the pace of kernel updates in Fedora led to breakage in these layered projects.
We formed a team I led at Red Hat to go about solving this problem, and, after approaching and discussing it with the CentOS Project core team, Red Hat and the CentOS Project agreed to “join forces.” We said joining forces because there was no company to acquire, so we hired members of the core team and began expanding CentOS beyond being just a rebuild project. That included investing in the infrastructure and protecting the brand. The goal was to evolve into a project that also enabled things to be built on top of it, and a project that would be exponentially more open to contribution than ever before—a partial solution to the openness gap.
Bringing home the CentOS Linux users, folks who were stuck in that availability gap, closer into the Red Hat family was a wonderful side effect of this plan. My experience going from participant to active open source contributor began in 2003, after the birth of the Fedora Project. At that time, as a highly empathetic person I found it challenging to handle the ongoing emotional waves from the Red Hat Linux split. Many of my long time community friends themselves were affected. As a company, we didn’t know if RHEL or Fedora Linux were going to work out. We had made a hard decision and were navigating the waters from the aftershock. Since then we’ve all learned a lot, including the more difficult dynamics of an open source development methodology. So to me, bringing the CentOS and other rebuild communities into an actual relationship with Red Hat again was wonderful to see, experience, and help bring about.
Over the past six years since finally joining forces, we made good progress on those goals. We started Special Interest Groups (SIGs) to manage the layered project experience, such as the Storage SIG, Virt Sig, and Cloud SIG. We created a governance structure where there hadn’t been one before. We brought RHEL source code to be housed at git.centos.org. We designed and built out a significant public build infrastructure and CI/CD system in a project that had previously been sealed-boxes all the way down.
However, the development of RHEL itself still remained closed behind the Red Hat firewall. This had been true for almost twenty years. For the open source development ecosystem this has been an important and often painful gap—it’s the still same openness gap as 2003.
This brings us to today and the current chapter we are living in right now. The move to shift focus of the project to CentOS Stream is about filling that openness gap in some key ways. Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL.
Just as when we joined forces, Red Hat approached the CentOS Project with its plan, and the CentOS Board signed on to it. That plan centered around not just closing the feedback-loop part of the openness gap, but in finding a way to help evolve RHEL development from happening inside of Red Hat to outside of it.
The Board was fully aware that in filling one gap we risked reopening the availability gap on the end-user side of the equation. While CentOS Stream would be open to contribution in a way that it never had been before, it would stand the risk of being somewhat different than CentOS Linux has been.
But we also knew as a project trying to do two antithetical things at once would mean doing both poorly. Providing our community with a solid, reliable distro that is good-enough for your workloads is a strong part of the CentOS brand. We’re confident that CentOS Stream can do this.
And while I’m certain now that CentOS Linux cannot do what CentOS Stream can to solve the openness gap, I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the end.
Beginning now is the time to genuinely help the CentOS Project understand what you need in a CentOS Linux replacement, in some detail. Even your angriest of posts are being read, and your passionate viewpoints are being seen and understood. I’m not the only Linux old-timer working on this.
This is your chance to be recognized for where you land in the availability or the openness gap, and how it is being there, so that the people crafting RHEL solutions are doing it with your use case(s) in mind. This input is happening right now. The new email address firstname.lastname@example.org goes directly to the people in the business unit (who are not in Sales) trying to solve your problems using this open source development method.
It is hard to balance the needs and processes of making business decisions with the needs and processes of making open community decisions. Arguably, Red Hat has been among the best organizations at straddling this hard, thin line. If you trust our code enough to run it for this long, I ask you to trust us to make good decisions here. I ask you to trust Red Hat and the CentOS Board to work with you to find a way to bring the community along into the next chapter.
If you want to talk with me further, the best place is the centos-devel list or Twitter.