CentOS Stream is Continuous Delivery

Friday, 11, December 2020 Stef Walter Uncategorized 65 Comments

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already 
as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Fedora, CentOS Stream and RHEL delivery

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL

65 thoughts on "CentOS Stream is Continuous Delivery"

  1. Ang says:

    No one has a problem with CentOS stream existing, what people have a problem with is discontinuing of CentOS 8 and breaching the commitments CentOS/RH/IBM made.

    This brings the question of how can CentOS/RH/IBM be trusted anymore? Which has yet to be addressed.

    1. That guy says:

      What commitment did CentOS make you? What were the terms of that commitment? Did you agree to something? If you chose to use CentOS Linux for dev/qa/prod that was YOUR choice to do so and no one is stopping you from doing that. RHEL customers have an agreement with RHEL where RHEL makes a commitment to them. CentOS came with no such thing. Whining about it only shows that people think they are entitled to something they actually are not. You don’t like the fact that CentOS Linux isn’t anymore? Good, then you can get the source binaries from RHEL, remove the branding, and maintain the release and spend ALL your time doing this so that others can be entitled to your distro. It’s your commitment to them right? That way other people can benefit and run production off of your releases without any insurance other than it’s some kind of commitment.

      1. Thing King says:

        > Whining about it only shows that people think they are entitled to something they actually are not.

        I suspect you are only looking at legal obligations and of course there are none. But to understand the person's POV you're replying to, it seems like you have not factored in the concepts of how goodwill and honest communication work in open source projects.

        Perhaps you look at this system as a one-sided arrangement: redhat gives and users take. But is that clear-cut? Where did most of the code come from in the first place? Don't centos users ask questions and find bugs? Doesn't broad use of centos lead to a great deal of software being ported to it, making it easier to install third party software on RHEL? Don't engineers studying RHEL download centos to practice for certification exams? Don't IBM sales engineers help centos users convert to RHEL customers with a special script? Hasn't redhat's participation in open source projects led to considerable positive reputation in the industry?

      2. Alex Mercer says:

        This. More people need to understand this. I am not at all angry with RH for this decision. At least, now people using Stream will have input on RHEL, which gives RH incentive to keep maintaining it and gives others a completely free continuous OS for near production-ready environments.

  2. Jan says:

    Can you please at least change the name? This obviously ain't got nothing to do with "Community" anymore.

    1. Paul W. Frields says:

      That's demonstrably false since with CentOS Stream, the community can actually do things like advocate for changes, do merge/pull requests, etc. And when they find something wrong, it won't take six months to see it turned around into a fix in RHEL. And of course it's still free as in both beer and freedom.

      1. Paul W. Frields says:

        To clarify: "turned around into a fix in RHEL, then wait for that fix to be rebuilt as part of CentOS Linux." Of course those doing the rebuilding continue to put in a fantastic effort on that. But not only can they spend that time on innovation now, but CentOS Stream will get that fix to consumers faster.

      2. Jan says:

        I was referring to the decision making process and the (dis-)respect of the board for the community in that regard.

  3. Peter says:

    Is there an official way to "transform" a CentoOS 8 Linux system into a CentoOS 8 Stream system so that it continues to receive updates from the Stream repos?

      1. Most of the users are now furious about this move despite deep assurances from the organisation. However, we need to accept the fact and move ahead. Those who still need to continue using CentOS will need to migrate to CentOS Stream from CentOS Linux. Worried users can check out the easy steps to migrate from CentOS Linux to CentOS Stream https://www.techsolveprac.com/migrate-centos-linux-stream/ featuring a video too capturing the live process.

        1. Ljubomir Ljubojevic says:

          Nope, all we will do is move to Springdale in the short term and to Lenix or Rocky Linux in the long term.
          After 12 years of using "CentOS Linux" (RHEL clone) I am NOT going to use "CentOS Stream" that at any time can brake my 3rd party drivers...

  4. Shawn says:

    Used the update to transform CentOS 8.3 to CentOS Stream 8 and no issues at all. All our software and panel ran fine.
    Used the instructions here: https://ostechnix.com/how-to-migrate-to-centos-stream-8-from-centos-linux-8/

    Will CentOS 8 stream seamlessly transition into CentOS 9 stream or is a full reinstall necessary?

    1. Stef Walter says:

      We're working on in-place upgrade from 8 -> 9. It's not implemented yet and is difficult engineering work given the combinatorial complexity involved.

      Here's what I know:

      a) It will certainly be more complex than the three commands upgrading from CentOS Linux 8 to CentOS Stream 8.

      b) It would look something like this: https://www.redhat.com/en/blog/upgrading-rhel-7-rhel-8-leapp-and-boom

      c) It's the kind of thing we can work together with the CentOS community on. In-place upgrades are difficult to get right without lots of feedback.

      This week, I had a discussion about CentOS Stream in-place upgrades with the architect of RHEL in-place upgrades.

      Given the frustration we've all had this week, I hope you understand that I'm trying to avoid stating something that can be seen as a commitment.

  5. John says:

    Is CentOS Stream supported for approximately 5 years? Then I think there will be no problem. And how often does the next version of CentOS Stream come out? ( 9.1 9.2 9.3...)

    1. Stef Walter says:

      Yes, it's supported for the Full Support cycle of RHEL. For CentOS Stream 8 that's 5 years, and I expect that for CentOS Stream 9 it'll something similar:

      https://access.redhat.com/support/policy/updates/errata
      https://centos.org/distro-faq/#q2-what-about-the-other-releases-of-centos-linux

      You'll notice that there are slight differences between the RHEL 7 and 8 lifecycles (see link above) ... so as we go to CentOS Stream 9 and 10 there may be again slight differences factored in. Such adjustments have always been present.

      Updates for CentOS Stream can come out as often as daily. There have been times when it has been updated weekly, due to CentOS infra or obstacles. CentOS Stream will smoothly iterate between 9.1 -> 9.2 -> 9.3 ... (and obviously the same for 8.x).

      Research has indicated that many CentOS Linux users never actually update until there's a security update. And then they have updated to the latest version to get that security fix.

      It may be that you update CentOS Stream with each and every change, or similarly wait until there's something you're need or are interested in.

      1. Chris Mair says:

        > For CentOS Stream 8 that's 5 years

        Or maybe not. Who trusts your EOL dates at this point? I don't.

        -- Chris

  6. cmdrlinux says:

    Don't break your arm patting yourself on the back there. Most of what you "invented" was already being done by other distros. Stop acting like you parted the red sea.

    If it's so easy why don't you continue to offer the support for CentOS 8 that you commited to providing? What a joke.

    1. Stef Walter says:

      I understand we're all frustrated. So I'll ignore the personal attacks.

      Will say it again. Yes things like (CI and CD) are obvious and routine techniques. Again, integrating *any* Linux Distribution is a grand challenge. Kudos to everyone who has integrated Linux in a modern way.

      I was not involved in the decision to stop providing support for CentOS Linux 8. Nevertheless, I can understand the anger of those who are being asked to switch to one of the alternatives.

      1. Ljubomir Ljubojevic says:

        Only alternative to RHEL clone is another RHEL clone. RHEL 8.3 kernel broke 100+ 3rd party driver modules from ElRepo. What happens when RH dev changes kABI on next kernel iteration and I run "dnf update" on remote system and loose storage or network? You come and fix it?

  7. Riot Nrrrd™ says:

    My take on this blog post:

    "Hey look everybody! We are now buzzword-compliant with all the kids these days! CI/CD! DevOps! Release early, release often!"

    That might be fine for the mobile apps world, where everyone is engaged in a bizarre nuclear arms race to release every week or two. It is completely unacceptable in the OS realm. It is literally *exactly the opposite* of what I want from my OS vendor.

    We are a mostly RHEL shop. But we were asked to admin a small project that is typical of Government Labs - small number of people, small number of systems, even smaller budget, so they can't afford RHEL licenses. They are standardized on CentOS 7.5 right now. Maybe by the end of next year they will be ready to consider migrating to something newer.

    When that day comes, I will have no story to tell them - their CentOS Linux is going away. Fedora is out of the question, RHEL is too expensive, and "CentOS" Stream is now a "perma-beta" RHEL and a moving target.

    1. Martin Pitt says:

      Do you literally mean 7.5? Is your project so brittle that it can't even keep up with minor releases and security updates? Then you will never be able to upgrade to a newer major version, and this has been dead for a long time, sorry. This is not a question of money or number of developers, but of methodology. But I hope you had just a typo there

      1. THJ2 says:

        Key word here is "government". The processes in the government space mean *many* thing run *years* behind, particularly on air-gapped networks that are perceived as "safe" prom hackers taking advantage of the blCVE of the day.

  8. Still very confused whether this is a positive move. But one thing is for sure that users have loose confidence in CentOS with this move. They may regain it after actually using CentOS stream for some time now on.

  9. Adam Woolhether says:

    Rocky Linux

  10. Linsoft says:

    The company I work for was speaking to redhat with a view to buying official licences and moving away from centos to RHEL.

    The behaviour shown with this move has actually stopped this happening as put simply, the trust is gone.

    development on transferring our product to centos 8 and ultimately RHEL 8 has now moved to oracle 8 where we will be buying in support with no transition of OS.

    1. Stef Walter says:

      Good feedback. If you'd like this feedback to reach the Red Hat business side you can email it to centos-questions@redhat.com

    2. Jamesnix says:

      My company is on the same boat. We are not interested in reaching out to RH business side either.

      1. Jan says:

        .... same here, too.
        I see no need in reaching out - trust is gone and I am certainly not going down the route RH tries to force us to go down.

  11. Roar says:

    I've been using Red Hat and derived OSes and CentOS for many years and have worked to make RH/CentOS the preferred OS in my company.

    The primary motive for my company using RHEL/CentOS on servers and CentOS workstations for developers are the stability and support from ISV and community. If it is supported on RHEL, it is working on CentOS. And because of small support resources it make sense to use the same OS family on servers and workstations.

    Developers are running one to several CentOS VMs to develop and test on. I run about 7-10 CentOS VMs. Paying RH Workstation/Developer licenses per VM per developer in addition to many RH Server licenses will cost a fortune to gain no more than CentOS give us today.

    If we cannot use CentOS for developer workstations any more and need to change to another distro (ubuntu/debian?) it also follows that the IT department will change server OS to the same distro family.

    I'm a SW developer and understand the reasoning behind CentOS Stream well and have no issues with Stream. As long Stream was living alongside standard CentOS.

    What I have issues with is that the CentOS board closing down standard CentOS releases. I'm not focusing on the shortened lifetime for CentOS 8, everybody outside RH/CentOS board understand why this was a stupid choice for the _user_ base.

    So why can we not use CentOS Stream instead of normal CentOS Release?

    For servers it would be stupid to run rolling relases on it. A server should run on a stable OS for years to come. Server should absolutely be updated to get security/stability fixes and so on, but going for a rolling release is taking it too far.

    Regarding workstation for our developers, can we use Stream there? Well, there are at least two issues I see for our use case:
    1) In a zdnet article for about a year ago, Red Hat CTO explain the reasoning for CentOS Stream and had these remarks:
    https://www.zdnet.com/article/red-hat-introduces-rolling-release-centos-stream/

    "To be exact, CentOS Stream is an upstream development platform for ecosystem developers. It will be updated several times a day. This is not a production operating system. It's purely a developer's distro."

    For ecosystem developers wanting to get ahead of RH/CentOS releases and test their soultions agains it, fine. For us non-ecosystem developers needing a stable platform and not taking part of the next RH/CentOS relase, we value stability and production ready state more that the ability to see new changes coming.

    So when the Red Hat CTO says Stream is "not a production operating system. It's purely a developer's distro", it is clear that Stream is the oppsite of what our IT department need and the opposite of what we non-ecosystem developers need for a workstation OS.

    2) A second reason why we developers in my company cannot use Stream is that we uses a lot of proprietary software products from other companies.
    These products are tested and released for a few supported OSes. Red Hat/CentOS releases are often one of the supported OSes. And often they test against a specific minor version of RH/CentOS. I have a program that uses CentOS 7.5 as a supported OS, but I run updated CentOS 7.9 on my workstations. The program complain about running on a unsupported OS, but it mainly works because of the compability between minor releases of CentOS. The next version of the program may support a newer version of RH/CentOS, but it may not be released in 6 months time.

    But will these companies support a rolling release when they today are behind on stable releases? No, they will not, and the moment CentOS Stream changes one package/library to a version that is not compatible in any way with the proprietary program it stop working. And it may take many months before the program support this configuration, but then CentOS Stream has changed another component and so on. I really cannot see proprietary sw supporting CentOS Stream at all, and that is also a show stopper for us using CentOS Stream.

    So all the articles in the world describing how well CentOS Stream is going to be, how well it fits in the Red Hat developer flow, and how it is not a beta release for RHEL will not fix these two issues for us users that really selected CentOS to get a _stable_ and long term system supported by proprietary sw vendors. Stream make sense for Red Hat, ecosystem developers targeting next RHEL-version, and any users wanting newer sw without the stability requirement, but not for the majority of CentOS users today.

    Examining the options ahead is difficult. Won't touch Oracle stuff, way to expensive to replace CentOS workstation/server with RHEL-solutions. If we are migrating away from RH/CentOS we might just as well consider Ubuntu/Debian instead of other RPM-based distros like openSuse.

    As things stand now CentOS is dead for me after many many years, and I think it is really sad.

    1. pit says:

      I quite agree with this comment. the problem will come from the integration of third party software often proprietary with a low update frequency. A lot of modding software requires a certain version of compatibility, it works with 7.3 and not with 7.6 (yes yes I swear). And unfortunately we won't be able to fight against all these problems that will come our way.

      Even if I have total confidence in the centos stream project when it comes to reliability. I will not be able to take the risk of questioning the various contracts on third party software which are insdispensable to my company.

      We will study during the next year, the question of the viability of centos steam by comparing it to the needs that are imposed to us by the other companies notatment with the rocky linux project which seems for the moment to be the best alternative to centos.

    2. Ljubomir Ljubojevic says:

      Springdale Linux, RHEL clone already exists. Rocky Linux clone is in preparation, and CloudLinux plans to publish RHEL clone "Lenix" also, should be ready around March 2021.
      And notice that CentOS Linux 7 will be supported until EOL in 2024 and there will still be support for CentOS Linux 8 for next 12 months, enough to chose your exit strategy smartly and without emotions.

  12. Piet says:

    So, has someone forked centos yet? Centos once filled a gap that they left open again now. It will only be a matter of time before someone else starts a "centos 2.0" project.

    And in the mean time, I have stopped buying RHEL licenses. I had just started working on building a system based on centos 8 for development and rhel 8 for production. I'm now looking into Ubuntu instead.

    1. Stephen Lynx says:

      Rocky linux.

  13. ferricoxide says:

    The thing that this whole decision seems to neglect is people that have or are moving their RHEL-based solutions to AWS/Azure/etc. As a consultant working with a number of RHEL-using enterprises, leveraging a mix of CentOS and RHEL in cloud was the most-sensible cost-solution. Having no OS-cost, CentOS is a 40% savings over RHEL per compute-hour. Given that leveraging RHUI-enabled AMIs gives you effectively zero recourse for support for that 40% cost-premium, implementing CentOS for Dev and RHEL for T&I and Prod has become a very common use-case across my customers.

    Unfortunately, Red Hat's decision to effectively unlink the commonality between the two OSes likely renders that work-flow no longer sufficiently the same for many of my customers' security groups' accreditation-processes.

    Red Hat's creating a gap in the "Dev→T&I→Prod" approval-flow that my various customers' DevSecOps plans are going to need to account for. Unfortunately, Red Hat's newly-created gap is being done without really providing any cost-relief for the Dev portion of those plans.

    Ultimately, this means the cost of development goes up by 40% per compute-hour ...or finding solutions that don't involve prototyping on CentOS (which, in turn, involves ultimately not deploying on RHEL).

    That Red Hat doesn't really think of cloud-hosted customers – other than when trying to sell OpenShift-based solutions – has been a long-standing problem. This is just another example of that organizational blind-spot.

    1. Ljubomir Ljubojevic says:

      To be fair, RH did announce some form of no-cost and low-cost licenses, but they will be know only in Q1 of 2021.

      On the other side:
      Springdale Linux, RHEL clone already exists. Rocky Linux clone is in preparation, and CloudLinux plans to publish RHEL clone "Lenix" also, should be ready around March 2021.
      And notice that CentOS Linux 7 will be supported until EOL in 2024 and there will still be support for CentOS Linux 8 for next 12 months, enough to chose your exit strategy smartly and without emotions.

    2. Karsten Wade says:

      Thanks for your comment. Your situation is absolutely understood as being one that needs a better solution, either in the project or by Red Hat. This is literally what centos-questions@redhat.com is for—people who are creating solutions to your problems are at that email address. They need to know your detailed requirements and situation, beyond a comment in a blog post. This is exactly the conversation you can be having, telling about your pain points and asking for whatever you think the business might provide you.

      1. Jan says:

        "creating solutions to your problems"... within one year? After users already invested about a year preparing for the CentOS 8 transition.
        I have no problem with RedHat shifting the focus, but the timeline is laughable and leaves the impression RedHat has zero understanding for their own and CentOS users.

  14. Troels Arvin says:

    I think that Linux is now so mature and good that it is sensible to rely on a stream distribution, and i think CentOS stream is actually a compelling distribution.
    I am still a bit nervous, though. How do you do continuous testing? Specifically, do you make sure to continuosly smoke test it with key like JBoss, SAP, Oracle, Db2, etc?

    My reason for asking: I once saw I point release of RHEL (7.2) break Db2, Oracle, some PostgresSQL installations, and others: https://www.ibm.com/support/knowledgecenter/de/SSFUEU_7.3.0/com.ibm.swg.ba.cognos.op_installation_guide.7.3.0.doc/c_op_ig_trb_linux_72_ipc.html
    That led me to think your automatic testing was insufficient.

    1. Ljubomir Ljubojevic says:

      Add to that that RHEL 8.3 kernel changed kABI enough to brake 100+ (all) ElRepo (3rd party) driver kernel modules. And RH devs are going to paly with kABI inside Stream, so if you have hardware not supported by RHEL kernel...

  15. Dima says:

    I don’t know why I’m still reading this blog. I switched to Oracle Linux yesterday.

  16. Bayu Sanjaya says:

    Ok, i have simple question. For a small - medium company that want to use red hat compatible as the main OS, especially for production, which OS that you will choose
    1. Centos Stream, or
    2. Redhat downstream like Rocky Linux/Oracle Linux?

    1. Anderson Zardo says:

      Can't wait: Oracle (but I don't trust this company, there are corpses where it passes)
      Can wait: Rocky (at least until Dec '21 you still have updates on CentOS, so I think you can wait Rocky or Lenix to be ready)

      Don't have problem with beeing a RHEL beta tester: CentOS Stream

      1. Ljubomir Ljubojevic says:

        Springdale Linux, RHEL clone already exists. Rocky Linux clone is in preparation, and CloudLinux plans to publish RHEL clone "Lenix" also, should be ready around March 2021.
        And notice that CentOS Linux 7 will be supported until EOL in 2024 and there will still be support for CentOS Linux 8 for next 12 months, enough to chose your exit strategy smartly and without emotions.

  17. Chris Jones says:

    It would well with scalable applications or containers. Vendors on the other hand need to know the version you're running and they know exactly what version of packages you installed for support. I like the idea, I disagree with the implementation. You could have kept Centos as is and created a new distro of Centos for rolling releases where you don't really about package version and the interactions of those packages and kept 8 within support life cycle. This just seems to me as a if you want that, use Rhel which is fine, but what about dev and test where we don't want or need support.

  18. Sean says:

    Yeah, thanks IBM! Buh-Bye.

    1. Anderson Zardo says:

      So long RH, and thanks for the fishes!

  19. Tony says:

    Sorry if this seems ignorant but it's a sincere question. For those using CentOS who are displeased with the change described, would Fedora be an acceptable alternative? And if not, why not? Thanks, my main goal here is deciding whether to proceed with my (personal, on my home general purpose laptop) transition to CentOS, switch to Fedora, or go another way.

    1. CentOS Stream is not very different than the current version of RHEL,

      How different is a RHEL 8.2 machine from a RHEL 8.3 machine? That is the type of difference we are talking about here.

      Durng the 8.3 RHEL release, Stream will be what is going to be rolled into 8.4, just a few months early.

      Most of the shared libraries are going to be compatible. Most 3rd party apps will continue to work.

      In fact, it would make sense for 3rd party apps to build on CentOS Stream .. so their products will work on the next version of RHEL when it is released,

      It would actually be counter productive for them NOT to create and build a repo against Stream, since it is provided specifically for people to see what will be in the next RHEL. Why would they wait and missing getting it out on RHEL release day?

      1. Ljubomir Ljubojevic says:

        Guys from ElREpo reported every single 3rd party driver from ElRepo made for 8.2 is not working on 8.3 kernel, ElRepo devs needed to rebuild them ALL.
        Enough of the difference when ZFS storage or NIC stops working?

    2. Anderson Zardo says:

      Fedora Server still exist? If yes, I Think is too bleeding edge even for a SO with "server" on the name.

    3. morsik says:

      I'm huge fan of Fedora and CentOS/RHEL (or at least I was until this strange move…), but - no Fedora for normal critical servers!.

      This distro is great for desktop, I'm using it for many many years without bigger problems, but for servers… 6month release cycle is the same as Ubuntu ones.

      It may be good for some Kubernetes clusters (where you don't really care about operating system because you upgrade nodes with kubernetes anyway), or for some CI nodes (because maybe you use GitLab CI and run all steps in containers anyway).

      If those are not the case - I wouldn't recommend Fedora as a real server.

  20. Stephen Lynx says:

    So, what about the 2029 EOL that was promised? How are you guys trying to spin that one?

    1. Ljubomir Ljubojevic says:

      Red Hat representatives and employers claim they never promised 2029 EOL, that what was posted on the wiki was some clueless CentOS volunteer...

      1. Miko says:

        CentOS is supposed to be a volunteer project, not a project run by a large corporation like Red Hat/IBM. Hell, it's even in the name; *Community* Enterprise Operating System. CentOS is no longer a community OS, but rather another testing ground for RHEL run by Red Hat.

  21. swa says:

    And this is what my CentOS 8 Stream machine said today:

    ===================================================================================================================================================================================
    Package Architecture Version Repository Size
    ===================================================================================================================================================================================
    Skipping packages with conflicts:
    (add '--best --allowerasing' to command line to force their upgrade):
    cups-filters-libs x86_64 1.20.0-22.el8 CentOS-LIC-TC-AppStream 135 k
    libreoffice-core x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 109 M
    libreoffice-data noarch 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 2.1 M
    libreoffice-graphicfilter x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 414 k
    libreoffice-langpack-en x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 176 k
    libreoffice-opensymbol-fonts noarch 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 216 k
    libreoffice-pyuno x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 313 k
    libreoffice-ure x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 2.3 M
    libreoffice-xsltfilter x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 445 k
    poppler x86_64 20.11.0-1.el8 CentOS-LIC-TC-AppStream 1.1 M
    poppler x86_64 20.11.0-1.el8 appstream 1.1 M
    Skipping packages with broken dependencies:
    cups-filters x86_64 1.20.0-22.el8 CentOS-LIC-TC-AppStream 779 k
    libreoffice-calc x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 8.7 M
    libreoffice-draw x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 95 k
    libreoffice-emailmerge x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 92 k
    libreoffice-filters x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 85 k
    libreoffice-impress x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 666 k
    libreoffice-math x86_64 1:6.4.7.2-4.el8 appstream 93 k
    libreoffice-pdfimport x86_64 1:6.4.7.2-4.el8 appstream 311 k
    libreoffice-writer x86_64 1:6.4.7.2-4.el8 CentOS-LIC-TC-AppStream 3.8 M
    poppler-cpp x86_64 20.11.0-1.el8 CentOS-LIC-TC-PowerTools 74 k
    poppler-glib x86_64 20.11.0-1.el8 CentOS-LIC-TC-AppStream 174 k
    poppler-utils x86_64 20.11.0-1.el8 CentOS-LIC-TC-AppStream 247 k

    Transaction Summary
    ===================================================================================================================================================================================
    Skip 23 Packages

    Nothing to do.
    Complete!

    1. Ljubomir Ljubojevic says:

      CentOS Stream will be ready, and stat getting updated packages only in Q1 of 2021 (till March 2021). Only then will the changes (and dancing with the stars) truly begun.

      1. swa says:

        I surely hope so, but you can imagine that I cannot just 'hope for the best' in these matters.

        Right now the kickstart (one that tries to install i.e. inkscape on a workstation) that worked fine 7 days ago, cannot be used with Stream for nearly a week now. This implies that I have no clue when exactly my deployment is broken if Stream does not get its CI/CD into shape and only add rpms to the repo if there are thorough checks (can be automated) on the dependencies and all! That is just basic stuff in CI/CD i'd say.

  22. Ljubomir Ljubojevic says:

    And what will do owners of hardware not supported by RHEL kernel when you brake kABI and they loose 3rd party driver (from ElRepo) because old driver module does not work with new kernel you just installed?

    Oh, I know, move to Springdale, Rocky Linux or Lenix (CloudLinux is investing more then $1 million/year in bug-for-bug RHEL clone)...

  23. Dude says:

    It's a horrible example of evolution - if by evolution you mean biological evolution. That is supposed to be an unguided process. It sure seems like some rational beings guide software development. In fact, some of them even do it directly themselves. 😉

  24. Lilian M. says:

    How I see it: management decided to find a solution to avoid the "boot hole" vulnerability issue again(broken systems from an enterprise product), so CentOS users are made the guinea pigs for the enterprise version.

    I just need CentOS to build software against it, to support older versions of RedHat, without having to pay to legally support their distro - doesn't break my use-case but still very shady(it's not an option that users can take, it's forced on the community).

  25. Jan van Westland says:

    From the Rocky Website:
    The goal of Rocky Linux is to become a downstream build just as CentOS had done before, by building releases after they are added by the upstream vendor, and not before (Rocky Linux is led by Gregory Kurtzer, founder of CentOS project.)

    BeyBey CentOS, Rocky here we come !!!!!

  26. Jan van Westland says:

    It's all part of the evolutional process. The CentOs stain will die out, and the Rocky strain will flourish.

  27. Andre says:

    I don'see mutch updates with Centos 8 stream... at least once a week a update, to keep things moving..

  28. Jay says:

    Can CentOS 8 stream be easily upgraded to CentOS 9 stream ? Or it will need fresh install?

Leave a Reply to morsik Cancel reply

Your email address will not be published. Required fields are marked *