These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.
I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.
I’ve been on the CentOS Project Governing Board since its creation. I also was part of the consensus decision that we recently announced about shifting the project’s focus. I’ve cared about this space for a long time, for my 19 years at Red Hat and prior to that. I was involved in the Fedora Project since the earliest days, leading the documentation project and sitting on the then-Fedora Board, among other roles. I led the team at Red Hat that brought the CentOS Project in closer to Red Hat in 2013/2014, and as a result of that work I earned a seat on the CentOS Governing Board, where I was the Red Hat Liaison and Board Secretary until Spring 2020.
Let’s go back to 2003 where Red Hat saw the opportunity to make a fundamental change to become an enterprise software company with an open source development methodology.
To do so Red Hat made a hard decision and in 2003 split Red Hat Linux into Red Hat Enterprise Linux (RHEL) and Fedora Linux. RHEL was the occasional snapshot of Fedora Linux that was a product—slowed, stabilized, and paced for production. Fedora Linux and the Project around it were the open source community for innovating—speedier, prone to change, and paced for exploration. This solved the problem of trying to hold to two, incompatible core values (fast/slow) in a single project. After that, each distribution flourished within its intended audiences.
But that split left two important gaps. On the project/community side, people still wanted an OS that strived to be slower-moving, stable-enough, and free of cost—an availability gap. On the product/customer side, there was an openness gap—RHEL users (and consequently all rebuild users) couldn’t contribute easily to RHEL. The rebuilds arose and addressed the availability gap, but they were closed to contributions to the core Linux distro itself.
In 2012, Red Hat’s move toward offering products beyond the operating system resulted in a need for an easy-to-access platform for open source development of the upstream projects—such as Gluster, oVirt, and RDO—that these products are derived from. At that time, the pace of innovation in Fedora made it not an easy platform to work with; for example, the pace of kernel updates in Fedora led to breakage in these layered projects.
We formed a team I led at Red Hat to go about solving this problem, and, after approaching and discussing it with the CentOS Project core team, Red Hat and the CentOS Project agreed to “join forces.” We said joining forces because there was no company to acquire, so we hired members of the core team and began expanding CentOS beyond being just a rebuild project. That included investing in the infrastructure and protecting the brand. The goal was to evolve into a project that also enabled things to be built on top of it, and a project that would be exponentially more open to contribution than ever before—a partial solution to the openness gap.
Bringing home the CentOS Linux users, folks who were stuck in that availability gap, closer into the Red Hat family was a wonderful side effect of this plan. My experience going from participant to active open source contributor began in 2003, after the birth of the Fedora Project. At that time, as a highly empathetic person I found it challenging to handle the ongoing emotional waves from the Red Hat Linux split. Many of my long time community friends themselves were affected. As a company, we didn’t know if RHEL or Fedora Linux were going to work out. We had made a hard decision and were navigating the waters from the aftershock. Since then we’ve all learned a lot, including the more difficult dynamics of an open source development methodology. So to me, bringing the CentOS and other rebuild communities into an actual relationship with Red Hat again was wonderful to see, experience, and help bring about.
Over the past six years since finally joining forces, we made good progress on those goals. We started Special Interest Groups (SIGs) to manage the layered project experience, such as the Storage SIG, Virt Sig, and Cloud SIG. We created a governance structure where there hadn’t been one before. We brought RHEL source code to be housed at git.centos.org. We designed and built out a significant public build infrastructure and CI/CD system in a project that had previously been sealed-boxes all the way down.
However, the development of RHEL itself still remained closed behind the Red Hat firewall. This had been true for almost twenty years. For the open source development ecosystem this has been an important and often painful gap—it’s the still same openness gap as 2003.
This brings us to today and the current chapter we are living in right now. The move to shift focus of the project to CentOS Stream is about filling that openness gap in some key ways. Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL.
Just as when we joined forces, Red Hat approached the CentOS Project with its plan, and the CentOS Board signed on to it. That plan centered around not just closing the feedback-loop part of the openness gap, but in finding a way to help evolve RHEL development from happening inside of Red Hat to outside of it.
The Board was fully aware that in filling one gap we risked reopening the availability gap on the end-user side of the equation. While CentOS Stream would be open to contribution in a way that it never had been before, it would stand the risk of being somewhat different than CentOS Linux has been.
But we also knew as a project trying to do two antithetical things at once would mean doing both poorly. Providing our community with a solid, reliable distro that is good-enough for your workloads is a strong part of the CentOS brand. We’re confident that CentOS Stream can do this.
And while I’m certain now that CentOS Linux cannot do what CentOS Stream can to solve the openness gap, I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the end.
Beginning now is the time to genuinely help the CentOS Project understand what you need in a CentOS Linux replacement, in some detail. Even your angriest of posts are being read, and your passionate viewpoints are being seen and understood. I’m not the only Linux old-timer working on this.
This is your chance to be recognized for where you land in the availability or the openness gap, and how it is being there, so that the people crafting RHEL solutions are doing it with your use case(s) in mind. This input is happening right now. The new email address centos-questions@redhat.com goes directly to the people in the business unit (who are not in Sales) trying to solve your problems using this open source development method.
It is hard to balance the needs and processes of making business decisions with the needs and processes of making open community decisions. Arguably, Red Hat has been among the best organizations at straddling this hard, thin line. If you trust our code enough to run it for this long, I ask you to trust us to make good decisions here. I ask you to trust Red Hat and the CentOS Board to work with you to find a way to bring the community along into the next chapter.
If you want to talk with me further, the best place is the centos-devel list or Twitter.
"This brings us to today and the current chapter we are living in right now. The move to shift focus of the project to CentOS Stream is about filling that openness gap in some key ways. Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL."
Another long-winded post that doesn't address the single, core issue that no one will speak to directly: why can't CentOS Stream and CentOS _both_ exist?
Because in absence of any official response from Red Hat, the assumption is obvious: to drive RHEL sales.
If that's the reason, then say it. Stop being cowards about it.
Redhat has no obligation to maintain both CentOS 8 and CentOS stream. Heck, they have no obligation to maintain CentOS either. Maintaining both will only increase the workload of CentOS maintainers. I don't suppose you are volunteering to help them do the work? Be thankful for a distribution that you have been using so far, and move on.
Users seem so happy about Stream that they might as well maintain no CentOS at all...
We might be better off if Red Hat hadn't gotten involved in CentOS in the first place and left it an independent project. THEY choose to pursue this path and THEY chose to renege on assurances made around the non-stream distro. Now they're going to choose to deal with whatever consequences come from the loss of goodwill in the community.
If they were going to pull this stunt they shouldn't have gone ahead with CentOS 8 at all and fulfilled any lifecycle expectations for CentOS 7.
Red Hat is not supposed to be a distribution maintained by Red Hat, but rather a distribution maintained by the community (*Community* Enterprise Operating System).
Honestly, if it were about eliminating the freebie clones to force people over to be paying customers... This wouldn't be the way they'd go about it. In fact, why even bother with CentOS at all if this were their intention all along? All of the packages that make up RHEL (and pretty much every Linux distribution if we're being honest here) aren't entirely licensed with copyleft licenses like the GPL.
There are plenty of critical packages where the upstream is licensed as BSD, Apache, MIT, etc. They have no obligation to release the source code with those packages. So many important packages like OpenSSL are licensed with permissive licenses that aren't copyleft. They could effectively kill off all of the RHEL clones just by releasing only the code where upstream is copyleft. In fact, even where upstream is copyleft, they only have to make the source code available to paying customers. Sure, a customer could just take the source code and make it public anyways. But that's beside the point. They could make it extremely hard to near impossible for clones to exist if they really wanted to.
So why haven't they? For all of the time that CentOS and other clones have existed for more than 15 years, why now?
People have this misconception that open source is all about making things available free of charge. Being open source is all about transparency and crowdsourcing contributions from the broader community. Being free of charge is merely a side effect. Red Hat contributes code to upstream projects all over, even as big as the Linux kernel itself.
There's a lot of fair critiques out there about this decision. This conspiracy levied against RH to make a quick buck isn't one of them.
Good post. People thinking that all open source software should be free of charge don't seem to understand that companies that help create and refine open source software into stable versions, with accountability, need to pay employees to do the work. Otherwise when the big security vulnerabilities come along, who will be accountable and provide fixes in a timely manner. Systems level software is the place where security breaches often happen since it is the gateway to the overall system; with a path to the databases and all the applications. Not paying someone for this accountability and so they can pay top quality technical people seems not so smart to me.
I hear your frustration; this blog post is intended to answer things for you.
> Another long-winded post that doesn't address the single, core issue that no one
> will speak to directly: why can't CentOS Stream and CentOS _both_ exist?
They are antithetical to each other.
If we try to continue doing both, we will do both poorly.
Only one of the two distros can handle the use cases for both distros. That distro is CentOS Stream.
CentOS Linux cannot handle the CentOS Stream use case. We tried that, it didn't work. We have run out of time trying that method. It's time to move on.
CentOS Stream *can* handle the use cases for a vast vast vast amount of what people use CentOS Linux for. I'm confident of that.
Everyone has been trusting the code for 17+ years, developed out of Red Hat and distributed by the CentOS Project. This is the same code base, the same people, and all we ask is that you trust us to continue creating a Linux distro that works for you.
"Only one of the two distros can handle the use cases for both distros. that distro is centos stream"
Well actually no.
You can say that centos stream can handle both use cases IF you don't count the fact that a lot of people use centos because of it's 10 year support cycle. When you put that into the equation, then centos stream can't handle that use case.
> what about free versions of RHEL?
> wont that keep people using RH?
Do you REALLY think that a lot of the community would trust redhat as much as before they got BETRAYED?!
Sorry, but that's a BS. CentOS Stream and CentOS Linux are not mutually replaceable. You cannot sell that BS to any people actually knowing the intrinsics of how CentOS Linux was being developed.
If the problem was in money, all RH needed to do was to ask the community. You would have been amazed at the output.
No, it is just a primitive, direct and lame way to either force "free users" to either pay, or become your free-to-use beta testers (CentOS Stream *is* beta, whatever you say).
I predict you will be somewhat amazed at the actual results.
Not talking about the breach of trust. Now how much would cost all your (RH's) further promises and assurances?
CentOS users don't want an experimental distribution.
As Whitebox/Tao/CentOS showed us starting 2003, a pristine respin of RHEL can be created.
CentOS is no longer willing or able to provide a pristine respin, so we will begin the Whitebox cycle again.
I am beginning to see why CentOS stream is needed (to facilitate outside contributions to RHEL).
Still, CentOS stream is a surprise for those who have invested in CentOS 8, who are understandably looking for the exits.
Do you blame them?
> They are antithetical to each other. If we try to continue doing both, we will do both poorly. Only one of the two distros can handle the use cases for both distros. That distro is CentOS Stream.
They aren't antithetical, but they are indeed orthogonal. The solution, as has been reiterated constantly during this, and suggested before, is two *distinct* projects.
"CentOS Linux, the RHEL rebuild project" is distinct from "a community-developed Enterprise Linux upstream" and this new thing should have been given a different conceptualization. Call it RedHat Timberwolf, or El-hide, or whatever, and let CentOS Linux, the thing installed on millions of systems out there, continue to exist and live out its EL8 lifecycle.
RH has chosen not to do this but is confusing the issue on why this decision is being made but not letting others (easily) take over the workings.
RedHat has no shortage of folks able to come up with new trademarks. The concept of confusing the meaning of "CentOS" and thus killing the rebuild is a complete red herring.
No one doubts that the existance of CentOS Stream is a good thing, nobody is complaining against CentOS Stream being a good, open way to get new code into RHEL.
People are complaining because you are suddenly killing CentOS 8 which has been released last year with the promise of binary compatibility to RHEL 8 and security updates until 2029. There is no comprehensible reason for this move, other than some sales people dreaming about selling more RHEL licenses to all CentOS users. The build infrastructure and repositories for CentOS 8 are already in place, all the hard work has already been dealt with. Just keep rebuilding packages until 2029 and feel free to take a different route for future major versions.
I don't think it will help RHEL sales grow. If you search the internet, you can see there many people written blogs about how to do X in centos, and many guys answer question about why X doesn't work for centos on ser verfault. They are the major reason why engineers choose RHEL because there is a community, they can find useful resources for most basic requirements
on the internet without ask and wait from the RHEL support team. Now most of them will switch to ubuntu which have a more promised futute.
That people who are focused on stability would move to a distro like Ubuntu that does not maintain a vast ABI with forward compatibility, makes little sense. That is an much less stable option...
Common agile practices teaches us that, to mitigate risk, you need to inform yourself.
I see a lot of uninformed people rushing to make decisions, when there isn't even any need to rush. CentOS Linux 8 lives til end of 2021 and 7 further on.
inform yourself!?!?! A month ago when I built a centos 8 server I was informed I wouldn't need to worry about shifting os's until 2029 now it's 2021.
No need to rush, end of 2021, how many hundred of server do you need to migrate? Seriously this is a cluster f.
Here is a more pragmatic take. Executives get large bonuses based on quarterly/yearly sales. This rushed to end centos8 will inevitably cause business to bite the bullet and buy a license. That will improve sales and therefore bonuses. The people who will profit are also the decision makers. That's a pretty normal logically take on a 21st century corporate practice in securing short term sales without regard for long term issues that may arise.
Another BS comment to comment on...
There are a BUNCH of companies that worked their butts off to migrate HUNDREDS of servers to the next version of centos, and you think that we are UNINFORMED?! YOU are the one being UNINFORMED!
OOPS SORRY MEANT TO POST ON SHANE GLEESON'S POST NOT YO POST SORRY
oops nevermind sorry
The problem is not an objection to CentOS Stream. CentOS Stream is fine and good. But it is not a fully-binary-compatible free alternative to RHEL, and CentOS is. Especially considering aspects of RHEL licensing on Power, going to RHEL to replace our CentOS park just isn't viable for us, and we've been testing out SLES as a result - and liking it a lot. This move is nudging us into migration mode.
Appreciate your understanding of CentOS Stream, thanks. And I definitely hear your concern about the long-stated fully-binary-compatible goals. I see this raised often, and the questions I have are:
1. How do you know that CentOS Linux is "fully binary compatible" with RHEL?
2. How do you know that CentOS Stream won't be "fully binary compatible" with RHEL?
I can't answer for the person you asked. I've regularly used packages meant for RHEL on CentOS. And I've regularly experimented with RPMs built for CentOS on RHEL when they didn't exists on RHEL.
In years past, I could count on seeing bugs in system service handling to surface the same way in both and for the workarounds to be the same.
There can always be bugs, but I understood this interchangeability to be a goal of CentOS. As a home Debian user, this is/was a major reason for why I considered RHEL/CentOS a better choice for work.
Should the community consider this interchangeability a supported goal of Stream?
You're still trying to convince us that this move is good for CentOS Stream.
Well, it might be true, but the fact is that literally nobody cares about Stream.
People loved and needed CentOS, not CentOS Stream.
CentOS coming to an end is unfortunate, but understandable. What is infusing, and cause for trust issues, is the removal of 80% of the support lifetime. Further, the ambiguous lifetime of Stream makes it hard for anyone to trust that, either. Red Hat has established a pattern of taking a good thing and destroying any faith placed in it. If a project is going to be proclaimed as good for everyone, then it needs to stay within it's scope. If a non-profit is to be acquired, then it needs to be stewarded to maintain its goals. Doing otherwise places you in the same predicament as Google: can a product be trusted to do is job, and be trusted to do it's job for a long time? No one knows.
For all anyone knows, CentOS stream will be sabotaged to be inferior in performance and/or stability, and perhaps walked back after a sales pitch for RHEL.
You ask us to trust you to make good decisions here, but you've done everything to destroy trust placed in you.
Thanks for your comment, I hear your feeling that you can't trust Red Hat in acting this way.
But this is a decision about an open source project, it is not a non-profit organization nor is it a product like ones that Google ends from time-to-time. I know it's a surprise and disappointment, but the dynamics here are quite different.
The idea that CentOS Stream would be inferior or even sabotaged makes no sense. CentOS Stream will complete all RHEL tests before being released. It's literally the same code that goes into the RHEL update. The RHEL Engineering team need all of this to work to do their jobs.
That's an interesting statement. I've been understanding Stream to be a rolling release. I take that to mean packages are being updated in real time and that there are no "checkpoints" or releases.
Are you saying there will be something comparable to a full regression occurring on the side prior to each updated RPM set getting into Stream?
I think that he was thinking this:
1. a few years pass by
2. centos waterfall comes out
(even MORE bleeding edge
than fedora)
3. centos stream gets chopped
This is all very logical to improve the participation in a very important product, RHEL, that runs a large variety of the most important infrastructures in the world, with a secure backing by Red Hat. As other people have hinted at in comments, it is also okay if Red Hat says that they know many for-profit companies were using CentOS in production and free-loading, in many respects, off of the work that the Red Hat team that builds and backs RHEL does. And that Red Hat believes it is smarter and more secure for those places to buy and use RHEL vs. an unsupported offering. Red Hat is a business after all. I agree with the other comment that Red Hat seems to be tap dancing around that point. Geez, look at the SolarWinds security breach, which I would think make people review the systems and management software they use. Go ahead and describe this as one of the goals of the move, unless it is not true. Exhale, and move forward !
I like the concept of Stream and had already converted most of my servers and VM's from 8 to Stream before I heard the news. While I would like to have the option of a fixed version for production reasons I trust RH in their QA of Stream and can understand the business reasons of not maintaining both also.
Pulling the plug from CentOS does not seem at all like RedHat. The best businesses and members of the community. RedHat has given back greatly to the community. However, when there is near unanimous objections from the community, you may want to reassess.
There are 2 big possible scenarios:
1. RedHat does not know how the community uses CentOS.
2. The community is misunderstanding/RedHat is explaining poorly.
I'd say this is 90% Scenario 1 and about 10% scenario 2.
You cannot repurpose a project that was originally independent and not be surprised that trust has been broken.
You could fix this by releasing CentOS name and logo back to the community or work with the community to have a Stream Branch for stability that the community (such as ROcky LInux) maintains. Otherwise, this looks like you bought the Grand Canyon and turned it into a parking lot for a Casino.
Curious when you say "community" what your definition is. I see that word used a lot, but people seem to define it differently. To me, it means the developers and related people (e.g. documentation) that contribute to the building or advancement of an open source project - like all the various people that contribute to the Linux kernel being the most widely known project. Or do you mean "users of the free software" as part of the community ? And are these two different in your view ?
Thanks
Hi Bill,
I was using community in a very broad sense. I can see how I could have been more specific.
I think I should mention that without RedHat, there would be no CentOS. However, I think that CentOS and Open Source Software in general have been very beneficial to RedHat. RedHat has become the de facto Enterprise linux because so many people can use CentOS to learn, experiment, and proof of concept.
Linux also began as a "community" (shared) project. RedHat has immensely contributed and deservedly profited. We need both Community Open Source Projects and successful Corporate Open Source Companies.
You raise some important points, and I definitely agree there are loud objections and unclarity.
I don't see any plugs being pulled from the CentOS Project; I see the opposite in fact, I see it being plugged into everything in a way it couldn't be before. This is an open source project deciding its future direction, which is to continue producing the same quality Linux we always have, but in a way we think is going to be better for everyone.
Your question about releasing the CentOS brand to be managed by another contributor community is a good question. The concern is, there is a strong association of the CentOS brand with the Red Hat brand; it cannot be so easily given away and disassociated. The amount of heat being poured on Red Hat in all these discussions is proof of that—people see the CentOS brand and Red Hat as tied together. People saying they need the CentOS distro to be a RHEL rebuild is proof of that association.
So the two brands are tied together, and we have a shared goal going back almost two decades to provide a Linux distro that is open, available, sustainable, and stable-enough. That is what we have been doing, and that is what we are going to continue doing. I hope as things unfold in the next few months, it will become more clear for you why we decided this was the direction to take for the project.
I would say you are illustrating that you see CentOS differently than many in the community. You see it as a feeder product of RedHat. RedHat's property. There was a community working on CentOS before Redhat's involvement that disagree.
It was to RHEL as perhaps Mint is to Ubuntu. I don't feel that's accurate though either. By comparison, Canonical provides a supported distribution of Ubuntu. But Ubuntu is also a freely available distribution that anyone can use or participate in, though Canonical clearly "more equal" than the other contributors.
With registrations, entitlements and other such things, RHEL is much harder to deal with for a community that do not want to pay the support costs. CentOS, as a downstream, was a community solution that helped mitigate those sorts of problems. I believe we will see the efforts of Rocky and others pick up the same role now. Unfortunately, I believe the outcome will be worse for everyone involved compared to before Redhat "pulling plugs" or "repurposing" a well known distribution.
And I believe there will be the same sort of name confusion that we see with OpenOffice and LibreOffice. People will have to explain that if you want the stable, RHEL X.X compatible "Community Enterprise OS" you now what "Rocky" X.X or whatever the successor of "CentOS as a downstream" ends up being. Because that's what people have meant by "CentOS" for years now. That may not be what Redhat wanted, but Redhat didn't start CentOS.
"and we have a shared goal going back almost two decades to provide a Linux distro that is open, available, sustainable, and stable-enough."
Where did they mention stable-ENOUGH?
From what I saw on the internet, it was basically a RHEL clone!
How the flip did all these misconceptions survive for 20 YEARS??!!!
ANOTHER THING:
"How the flip did all these misconceptions survive for 20 YEARS??!!!"
Wasn't this a COMMUNITY ENTERPRISE LINUX DISTRO?!
Karsten, I see most of what you have written in the comments section as well as your OP as damage control to an upset community that feels betrayed for a multitude of reasons. The reasons for killing off the traditional versioned releases of CentOS are not to benefit the community, and saying otherwise is smearing lipstick on a pig. Even if that is an erroneous belief, this is not what the majority of the CentOS community believes and convincing them otherwise would be futile at this point as this whole announcement and switchover to Stream has been extremely poorly executed.
I don't think Red Hat predicted this amount of fallout, and it is my belief that Red Hat executives and employees should at this point in time have a meeting/have many meetings to discuss the possibility of retracting the recent corporate decision to kill off the versioned releases of CentOS which is causing irreconcilable differences between the community and Red Hat as to what the future of CentOS should be. The fact of being concerned with giving away the CentOS brand as you believe it's strongly associated with the Red Hat brand, when Red Hat has already tarnished its image by pulling the rug out from underneath the community, is a bit shortsighted. Nobody ever said they demand the CentOS brand name be associated directly with RedHat, and to be honest, it seems to me that you are trying to use a fictitious belief with no evidence to essentially hoard the CentOS brand name. Did you ever do any studies on this or do a survey on the community if they would even care? Please stop making up excuses and reasons for unpopular decisions and take some responsibility and listen to your community, rather than try to blog your way out of a mess trying to find every buzzword possible to attempt to pull the wool over the eyes of the community.
As an analogy, think about how many forks have came across over the years, with users switching to those software titles and not thinking much more about about where it was forked from. Users always adapt to new software title names as the result of a fork. The bottom line is that your users don't like the idea of being shoved into a pool of beta testers; for that there is Fedora. And I also get the aspect that the community wasn't giving back pretty much at all any code changes, so I am partially on your side but there needs to still be two versions of CentOS and the users can choose which version they want to use. If Red Hat is concerned that if users are given a choice that the majority will not run stream, then that is not a good reason to proceed forward and your existing community is going to continue to fragment and move to forks. Remember, only fork when there are irreconcilable differences!
I think Stream is a really nice idea. I am not sure that calling it an "upstream" is technically correct though, it still feels like it is downstream from RHEL. Since you wouldn't publish anything in Stream that you wouldn't use in RHEL. But that is just naming.
It is a bummer you decided to no longer produce non-rolling, stable binaries, but as long as the git repo has tags for the stable releases it can be up to the community to maintain binaries for those. The real sticking point is security updates. But I assume those will still be made to the stream sources. And so can be backported if needed.
I think it would be great if you would be working together with other projects like rocky, lenix, clearos and springdale. If only to make sure the tags are in place in the git repositories and alert them of any security updates. So they can concentrate on the availability gap that the traditional centos binary releases tried to close. Then it is clear CentOS is now about filling the openness gap and some other project is about filling the availability gap. Making them have different names might also make the purpose of the different projects more clear.
It is going to take a while for all of us to get used to how the code flows, but CentOS Stream is definitely upstream of RHEL in a classic way—code is maintained in the upstream rather than carried as patches in the downstream.
I think your observation about the git repo and security updates is astute.
I've already seen people reaching out between those projects you mention and people who work in CentOS Project and/or on RHEL. I'm sure the relationships will be collegial and mutually beneficial.
One point I have to disagree on is around the availability gap and the idea that only a rebuild of RHEL can fill that gap.
In all that I've read and heard, I see people being afraid that CentOS Stream won't be able to replace CentOS Linux for their needs. I've read about a lot of clever detective work and amazing testing and validation people do around the common convention of a RHEL point release. But it's too early for anyone to say with any level of certainty that CentOS Stream cannot actually fill that availability gap.
Which is why now is the time for people to use the centos-questions@redhat.com process to explain (in detail) why they believe only a RHEL rebuild will work for their need.
Really, you shouldn't try to "explain" anything. Any attempt to justify will on;y make CentOS governing board look worse.
In centos@ and centos-devel@ CentOS mailing lists there are many threads where people with deep knowledge of CentOS development intrinsics explain why CentOS Stream may never be a replacement for CentOS Linux.
Stream has its uses, of course. But it can't be a valid replacement for CentOS Linux.
I suppose the majority of CentOS community will finally migrate to emerging RHEL clones, that's all.
If that was the Red Hat / IBM goal - well, they will reach it all right.
To: centos-devel@centos.org
To: centos-questions@redhat.com
Hi,
Re: https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platform/
you can spin this to the moon and back. The fact remains you just killed CentOS Linux and your users' trust by moving the EOL of CentOS Linux 8 from 2029 to 2021.
You've alienated a few hunderd thousand sysadmins that started upgrading to 8 this year and you've thrown the scientific Linux community under a bus. You do realize Scientific Linux was discontinued because CERN and FermiLab decided to standardize on CentOS 8? This trickled down to a load of labs and research institutions.
Nobody forced you to buy out CentOS or offer a gratis distribution. But everybody expected you to stick to the EOL dates you committed to. You boast about being the "Enterprise" Linux distributor. Then, don't act like a freaking start-up that announces stuff today and vanishes a year later.
The correct way to handle this would have been to kill the future CentOS 9, giving everybody the time to cope with the changes.
I've earned my RHCE in 2003 (yes that's seventeen years ago). Since then, many times, I've recommended RHEL or CentOS to the clients I do free lance work for. Just a few weeks ago I was asked to give an opinion on six CentOS 7 boxes about to be deployed into a research system to be upgraded to 8. I gave my go. Well, that didn't last long.
What do you expect me to recommend now? Buying RHEL licenses? That may or may be not have a certain cost per year and may or may be not supported until a given date? Once you grant yourself the freedom to retract whatever published information, how can I trust you? What added values do I get over any of the community supported distributions (given I can support myself)?
And no, CentOS Stream cannot "cover 95% (or so) of current user workloads". Stream was introduces as "a rolling preview of what's next in RHEL".
I'm not interested at all in a "a rolling preview of what's next in RHEL". I'm interested in a stable distribution I can trust to get updates until the given EOL date.
You've made me look elsewhere for that.
-- Chris
I have no problem with stream. As mentioned before the issue is we were told we would have support as long as RHEL 8 had support and that was pulled out without any warning. That is the huge issue. If it started with RHEL 9, great! We can see how it is, we can evaluate how it works before we use it. We no longer have that option.
Well, you have at least few months to evaluate it all with calm.
Years, if you're not paranoid about stream 8.
I guess I was not clear on my point. CentOS stream is rolling (within the branch) from upstream and from my understanding won’t be as stable as basing it downstream from RHEL proper. Do I believe it will be stable? Sure. But I have to prove it and it will take more than a few months to prove it. it isn’t the same Distribution as before so it will require lots of testing no matter what to prove it’sstability.
And again, having support as long as RHEL 8 and then not having it hurts trust. If this happened for RHEL 9 I would have no problem. It happened after a commitment was made which hurts.
I guess my biggest issue is They should have announced this at the START of CentOS 8.0. Instead they started CentOS 8 with the belief it was going to be like CentOS7 have a long supported life cycle. What they did was basically bait and switch. Not cool. Especially not cool for those running multiple nodes on high performance computing clusters.
Why did Red Hat need to use CentOS to fill this pre-RHEL gap? Give CentOS back to the community and continue to donate to it. Why not then make something like Fedora-RHEL to fill that "stream" gap. Red Hat didn't need to stir up the community and further ruin trust by gobbling up CentOS. You're in the open-source software market, where it's not as simple as dollars and cents.
I think there’s no reason to further justify RH’s choice, since this post is basically a rewording of the previous posts.
These posts clearly explain (and justify) the value that CentOS stream has for RHEL paying customers and developers. Actually, it is also clear that the only need that has not been taken into account is the need of current CentOS users. From a financial point of view, it makes sense. RH is a (for-profit) company, therefore it does only what it is in the best interests of its (paying) customers and shareholder. If this interest contrasts with the interest of CentOS users, the first one will always prevail. And that is exactly what happened.
I believe that RH is very well aware that CentOS stream serve completely different needs than CentOS, thus the vast majority of CentOS users simply cannot switch to stream. Therefore, it is inevitable that most of CentOS users either buy RHEL or switch to another distribution. I believe that it is quite meaningless to continue discussing on this.
I have been deploying computer systems with CentOS for over fifteen years. One of the nice things about CentOS was it lagged behind new releases of RHEL and was as stable as a rock.
Why is that important? My institution has learned from experience to try to stay away from the betas and first releases of software products and wait for mature stable releases. Making the new CentOS a beta stream for RHEL results in a less stable operating system and increases our risk in providing high-performance computing services. Also, our
vendors for high speed interconnects, parallel file systems, graphics device drivers, and other third-party software may drop support for CentOS because of new fluidity and difficultly with testing a particular release.
You ask why this pragmatic attitude? Our small staff maintains about fourteen thousand (14,000) CentOS high-performance computing systems to support thousands of researchers across the United States. This change to CentOS's stability will greatly affect how our institution uses your product in the future.
Disclaimer: My opinion is my own and no one else's.
Only a rounding error number of people installed CentOS intending to use it as a beta version of RHEL.
We are running centos because it is a stable 10 year release of linux.
The fact that you can't see this is an incredibly problematic breach of trust.
I have over 300,000 Centos nodes that require Long term support as it’s impossible to turn them over rapidly. I also have 154,000 RHEL nodes. I now have to migrate 454,000 nodes over to Ubuntu because Redhat just made the dumbest decision short of letting IBM acquire them I’ve seen.
Whitehurst how could you let this happen?
Nothing like millions in lost revenue from a single customer.
That seems just a lie... some company with 454K nodes and requirements for long term support (therefore mission critical workloads) would have a considerable team of sysadmins, with enough manpower to build and maintain a Linux distro of their own...
I used to have an centos 8 system. I just edited some files under /etc/yum.repos.d/ and ran "dnf distro-sync". Now I have a Springdale system. Goodbye centos whoever you were.
Please get rid of systemd...
systemd's not going anywhere; there's nothing wrong with it.
Hi there,
here's my five cents on this:
In the past Fedora was upstream, CentOS was downstream.
Yes, CentOS was always "lagging behind" a bit, but it was a deliberate and informed decision of those users preferring a somewhat more conservative and reasonably paced Linux flavor. They deliberately chose to wait for changes making their way from Fedora through RHEL into CentOS which indeed could take months or sometimes years. But that was a known fact and accepted.
That has worked well for many years. For me and my company the core value of CentOS was its stability promise paired with a reasonable pace of progress, combined with its suitability for desktop use.
CentOS-8 (in fact RHEL-8) gave up desktop usability almost completely, but that could have been worked around somewhat. Still, it was frowned upon a lot. Now, by removing CentOS-8 in favor of CentOS-8-Stream, it is also losing its stability promise, thereby rendering it mostly unattractive for us.
Personally, I would not mind paying for an acceptably priced RHEL if it would not completely ignore serious desktop users' needs. (I did so in the past and was fine with that). But I do not see anything indicating that this is going to happen soon.
So sadly, after having used CentOS since its 4.x days on servers and desktops, I do not see a promising future for me and my company there.
Still, having enjoyed CentOS for about 15 years, I wish you good luck and success with 8-Stream.
Best regards,
Michael
Thanks for your comment, I wanted to address a few things.
CentOS Stream is going to be released as a rolling distro, but no one is requiring your company to consume it that way. It's the same Linux in CentOS Stream as in RHEL as in CentOS Linux. You can make the same choices about when you consume it as you have been.
The purpose of the centos-questions@redhat.com team is to take in things like your suggestions about an acceptably priced RHEL. It's not a sales lead generator. It's a space for sysadmins to share the real truth about why they need CentOS Linux or what is blocking them from using RHEL instead.
Just migrated to opensuse.Rather than crying for dead os it’s better to act yourself.Redhat is a sinking ship it probably want last next decade.Legendary failure like ibm never have upper hand in Linux world.It’s too competitive now.Customers have more options to choose.I think person who have take this decision probably ignorant about the current market or a top grade fool.
Centos 8, and Stream too when I updated to it, are broken for something as simple as providing libuv-devel matching the libuv package via dnf.
I had to rebuild it from the source rpm... something bigger seems to be wrong with current CentOS than Stream or not.
Sorry about your current experience, I encourage you to file a bug report about it. I suggest you use the CentOS Stream component at bugzilla.redhat.com, so a RHEL engineer can look at it.
The process of CentOS Stream taking over as the upstream for RHEL is underway, and is expected to be there for CentOS Stream 9. The current situation has many transitional elements to it.
IBM/RH/CentOS keeps replaying the same talking points over and over and ignoring the actual issues people have. You say you are reading them, but choose to ignore it and that is even worse!
People still don't understand why CentOS stream and CentOS can't co-exist.
If your goal was not to support CentOS 8, why did you put 2029 date or why did you even release CentOS 8 in the first place?
Hell, you could have at least had the goodwill with the community to make CentOS 8 last until end of CentOS 7! But no, you discontinued CentOS 8 giving people only 1 year to respond, and timed it right after EOL of CentOS6.
Why didn't you even bother asking the community first and come to a compromise or something?
Again, not a single person had a problem with CentOS stream, the problem was having the rug pulled under their feet! So stop pretending and address it properly!
Even worse, you knew this was an issue, it's like literally #1 on your issue list "Shift Board to be more transparent in support of becoming a contributor-focused open source project"
And you FAILED! Where was the transparency?!
A link to the issue:
https://git.centos.org/centos/board/issue/1
What a piece of stinking BS. What is this "gap" you're talking about? Nobody in the CentOS community cares about this pre-RHEL gap. You're trying to fix something that isn't broken. And doing that the most horrible and bizzarre way imaginable.
To be fair, Stream does service a purpose. As I understand it, work on the next version of RHEL (think RHEL9) happens on a public branch of fedora. Prior to Stream, there was nothing similar for fix releases of RHEL (think 8.4). This lets them move development of fix releases from a "Cathedral" model to more of a "Bazaar" model. I would say that is an improvement.
I'm still trying to be open minded and allowing Redhat the opportunity to convince me that their version of a rolling release is going to be more stable than common sense and years in the software industry would otherwise lead me to believe.
I believe an old gap is being recreated and that groups like Rocky will fill it. I''m not sure anyone will be better off for that.
What a bad decision!
Difficult to build an ecosystem, but it is easy to destroy.
As I understand it,
Fedora - RHEL - CENTOS just becomes
Fedora - Centos Stream - RHEL
Why just call them RH-Alpha, RH-Beta, RH?
Anyone who wants to continue with CENTOS? Fork the project and maintain it yourselves.
That how we got to CENTOS from Linus Torvalds original Linux.
i thought fedora was the upstream project to redhat? what does centos-stream do that fedora can't do?
Fedora is able to move quickly to keep focused on the pace of Linux innovation.
RHEL Engineering picks a particular point in Fedora to branch from, which becomes the RHEL major release. That major release until now has been developed as a RHEL branch, and after this will be developed as a CentOS Stream branch.
All the stabilizing and other work that goes into the RHEL branch will now go into the CentOS Stream branch on its way to RHEL.
Meanwhile, Fedora continues upstream doing rapid innovation.
I can only comment this as dissapintment, if not betrayal, to whole CentOS user base.
This decision was clearly done, without considering impact to majority of CentOS community use cases.
If you need upstream contributions channel for RHEL, create it, do not destroy the stable downstream.
Clear and simple. All other 'explanations' are cover ups for real purpose of this action.
This stinks of politics within IBM/RH meddling with CentOS.
I hope, Rocky will bring the desired stability, that community was relying on with CentOS.
Goodbye CentOS, it was nice 15 years.
All this talk and just 1 mention of "money" in this thread.
Money is the motivation. A portion of CentOS users will migrate to RHEL, some users will become CentOS beta-testers and some other users will migrate away from CentOS. The outcome at this point is additional profit. Over time the beta-testers who couldn't afford RHEL and run production on CentOS-Stream will have had one burning production issue too many and will eventually either migrate to RHEL or to something else. The outcome at this point is still additional profit. But the outcome over a longer period because CentOS no longer holds the stable-and-free crown the gap will be filled by either something that exists, or a new distro (rocky?).
In the end it seems the time to say farewell to CentOS has arrived. I don't see any reason to use CentOS-Stream at all. So long and thanks for all the fish!
We've just agreed to cancel out RHEL subscriptions and will be moving them and our centos boxes away as well. It was a nice run but whike it will be painful, it is a chance to move far far away from the terrible decisions made here.
Cordially,
I would like to comment on that decision.
5 years ago I started looking for a distribution among all possible ones that would answer the following question. What I need on a Linux server that doesn't exist on Windows.
I need stability, long support time for security breaches and a distribution that has an (active community *). When you choose a distribution I believe that many people take this into account, because when you need to install something, the more likely you are to get support or help in a system that is used on a large scale.
Well, I chose CentOS and recommended during that time its use due to these qualities (community support - stability - perfect system). I chose a distribution that would go online 365 days with little chance of error.
The big difference with some distributions is that some update and create new versions every 3 years and 5 years and others that take time to create new versions like CentOS and continue to run smoothly and with security support. In my opinion the second characteristic is the best.
As an example, you can run a website by installing a simple php + mysql + apache on any distribution. However, if you want a free hosting panel you can install VestaCP (I recommend) or pay for one that I also consider one of the best (CPanel). See the big problem. These programs do not always update constantly following the various versions released on Linux. As an example, VestaCP is still only compatible with CentOS 7. As for CPanel, almost all based on CentOS recommends version 7, with version 8 still experimental. He realizes that not always in terms of compatibility is linked to the speed of development.
I read a lot of information about that note about this change, well, I believe it was a bad decision without thinking about the real problems that would imply. I honestly have some questions.
1. If the problem is financial collection for payment of staff, support, they could have placed a donation on the main website or charge 1 dollar or euro for the license and use CentOS. Monthly or yearly. It would be a way of raising money and I believe that many people would tend to contribute. I wouldn't mind paying $ 12 annually to continue CentOS.
2. Why not think backwards then? Why not put new developments and updates on Red Hat (Beta) and then migrate to CentOS. Because it would cause problems in Red Hat and you know it, but how it gets paid would not be possible to do so.
3. The big problem of the breach of reliability, yes! There is no denying that if you launch a CentOS 8 with support until 2029 and unilaterally extinguish by 2021 you lose all confidence in the system. What did you think that would not happen?
Even though many people are looking to pay Red Hat for an OS, unfortunately I believe that many people have stopped using CentOS. And that will also affect Red Hat as the two always went together. Finding a bug in CentOS was beneficial for Red Hat, now imagine you lose thousands of users, communities, repositories. It will indirectly affect both systems.
Honestly, you could have opted for a side project with CentOS Stream and CentOS Linux, even maintaining version 8.
When you make risky decisions like that, you should probably know the risks and the consequences. Even worse if they don't know.
I honestly do not know if it is possible to reverse all that has been done, either by releasing CentOS to remain independent, going back to those decisions.
Unfortunately in my case I think about looking for another distribution. Whether it is 3 years (debian) or 5 years (ubuntu) support. I believe that unfortunately it will be the direction of thousands of people. Look, I still have 7 installed with support until 2024.
It is not what I would like to do, abandon yum. Asking to believe that CentOS Stream will be stable is a little tricky. Although I believe in your ability.
Well, there were some considerations, maybe you can still reverse this situation before the boat completely sinks. One suggestion is to continue with 8 until 2029 and perform Linux Stream in a side project. I think it is not a shame to go back on a bad decision, a shame is to see CentOS end and nothing to be done. Unfortunately, I believe this will happen if the course continues like this.
Sincerely!
Karsten - I'm using the 30 day trial subscription to Red Hat Enterprise Linux 8.3. 8.2 didn't install - I eventually managed to file a bug (which was fixed in 8.3 and noted in the release notes.) On self-support, I'm not able to file a bug but 8.3 doesn't play nicely either, not allowing you to register the system. I'm worried about the amount of testing that RHEL is getting - and that's before Streams feeds in some degree of instability. Who can I talk to??
Anyone reading this comment, go join rocky Linux, help and contribute otherwise go with the likes of Ubuntu / Debian. Stop using CentOS it no longer has credibility over its name "Community enterprise Linux", nothing community about what they just did. From the article there's a sense of dictatorship and no respect nor apology for cutting CentOS 8 support down to 2021, just waffling nonsense. Believe me you don't want to do business with these kind of organisations. If money was an issue for them, I and many others would've happily donated, in fact many businesses would've paid donations. Just remember because of the likes of CentOS community Redhat remained relevent in the market, with hundreds if not thousands of users reporting bugs and fixes, the community at large played a huge role bringing about more stability to Redhat OS. What you've done is pooped all over this and pushed many to seek alternative routes. If the likes of Rocky Linux doesn't gain traction and everyone hops to Ubuntu, you can kiss IBMs Redhat Linux goodbye cos it really won't survive.
"It gives the CentOS contributor community a great deal of influence in the future of RHEL." (from December 8 post)
"RHEL users . . . couldn’t contribute easily to RHEL." "The goal was to evolve into a project that . . . would be exponentially more open to contribution than ever before".
What is the full scope of this increased level of participation? How much influence will the community really have in shaping the next RHEL release? Stream has been described by some as essentially a preview of the next RHEL point release, so I am genuinely curious how much influence the community would really have.
Another IBM acqui-kill.
Congratulations.
Look At This Reddit Post Title:
"Straight from the horse's mouth: "CentOS was not providing that much usefulness to Red Hat""
No Wonder They Cut Off CentOS Linux,
They don't even care too much about the project! (CentOS board obviously cares much more tho, so don't shoot me down about that)
Karsten, address the EOL date shift. That's the pain point for many, and why we no longer trust RHEL worth a damn. If you can't/won't, you're wasting your time and CentOS will die with Stream as a casualty of IBM's incompetence and greed.
I must say I am sad to see CEntOS go. For a Rolling Release we got arch. Let's see if there will be any alternatives.
Looking at this from both sides, it seems that all the benefits (if anybody even cares about 'stream') go directly to IBM aka Redhat. They have yanked out the rug from underneath the many users, including those in enterprise environments that can take 8-10 months or more to implement new environments.
Why not just leave Centos alone? I'm just an individual, but have already spent many hours converting my Asterisk-PBX platforms to Debian because I obviously can't rely on this experiment to be the foundation of something that requires solidity and reliabilty.
Who wants to develop for a wealthy company for nothing? Go rockylinux, go!
Meantime back at the ranch; https://www.wsj.com/articles/ibm-hopes-to-double-sales-at-red-hat-in-next-three-years-11603213076?cx_testId=3&cx_testVariant=cx_22&cx_artPos=4#cxrecs_s
In the beginning, there was Linux. A free OS. An oddity coming on the heels of the Unix Wars (Solaris vs. HP-UX vs. AIX vs. Tru64Unix, etc), but it had promise. Loads of folks put in their time to make it a great OS. RedHat was formed, taking this free/opensource OS and building the standard opensource support model. The bits are free, but if you want support, you pay. But now Linux is so big, so pervasive, that the biggest baddest traditional computer company, Itty Bitty Machine Company (IBM) grabs RedHat and says "no more of this free stuff... we can make a LOT of money". IBM didn't pay billions for RH to give the bits away for free.
So they now say to all of us, no more free bits (CentOS), pay for RHEL, or else. Forget that it is opensource. Forget that it was built to be free. Spinning it any other way is disingenuous. It is a money grab.
Hopefully this move will not result in the fracturing we saw with Unix, where each 'distro' went their own way, and Solaris competed with HP-UX, AIX, Tru64, etc. and app developers gave up in frustration. I can see the same thing happening with the RHEL build vs Debian vs SUSE. The kiss of death.
Here you can read the truth: https://www.theregister.com/2021/01/26/killing_centos/
"Brian Exelbierd, responsible for Red Hat liaison with the CentOS project and a board member of that project, has told The Register that CentOS Linux is ending because Red Hat simply refused to invest in it."
The truth is, Red Hat did not achieve its goals with the CentOS Project. Now it's just being killed. Regardless of the consequences.
I work for a not-for-profit that has very real, federal, security measures in that have to be in place. We don’t need just need stable, we need rock freaking solid. We already have a large Windows environment. It’s expensive and there simply isn’t anything we can do about it considering the field we are in. We have a few RHEL Linux boxes and licenses, but the bulk of our Linux inventory is CentOS because we are not-for-profit. This comes at a bad time for certain as I actually just began provisioning new servers for the purpose of migrating several non-critical systems over to CentOS 7 and maybe even start to automate by replacing some of our shell scripting with Ansible. We had also started to evaluate whether RHEL 8/CentOS 8 was stable enough to begin transitioning some of our more critical systems from RHEL 7/CentOS 7 over to their respective 8 counterparts. We have halted these projects as we contemplate what we need to do now, and I have been instructed to look for viable alternatives. Fortunately, we had not transitioned anything over to CentOS 8 or RHEL 8.
As a Linux lover, it pains me that this has occurred, because it might just be the end of Linux use at my facility. I am not sure that the “powers that be” will want to move to another Linux distribution and I have already began to hear the cringe-worthy words “Will it run on Windows? If so, why not just move it there?” I’m not sure how many other people are in our unique situation, but whether anyone wants to admit it or not, Windows isn’t going anywhere anytime soon. Since most end-users are familiar with Windows, it’s difficult to get them to learn Linux, or even Mac OS. This unfortunately means that anyone else is competing for what budget finances are left over, and in a world that is still dealing with a COVID-19 pandemic, it’s not much.
I get the business move that is forcing companies to pay for RHEL. Non-corporate end-users aren’t going to do it, there’s too many “stable enough” distros out there. With CentOS beating the pants off of RHEL in terms of Linux server OS (last I saw Ubuntu was ~47.5%, CentOS was ~18.8%, Debian was ~17.5%, and RHEL was a distant ~1.9%), I totally understand why RH wants RHEL to be the only stable distro in its…well…stable. If you are giving away stable, you aren’t making money. While I get it, I think there is a serious lack of understanding of the amount of weight RH can throw around in relation to where they are on the food chain. I could be wrong there, but I think when push comes to shove, there are just already cheaper ways to go about implementing things with other resources already in place. When it comes to going up against Windows in terms of budget, whoever had better be bringing something to the table that is a need that Windows can’t fulfill, or they’re just going to lose by simple numbers. It’s a hard truth, but truth nonetheless.