How RHEL is Made

Friday, 11, December 2020 Brendan Conoboy distro 21 Comments

This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time.  CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified.  Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?  If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built.  Let’s get into it!

Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto.  Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch.  Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago.  While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.  

Let’s start with grounding: every RHEL minor release is based on the previous release, plus targeted backports of upstream development.  Often, Red Hatters are the original authors of those patches, but there are no shortcuts: upstream acceptance is the first test every patch must go through before we start it through the journey that eventually leads to a patch’s integration in the release.  Even then, this is about an upstream patch existing, but that alone will not guarantee a patch’s inclusion.

Any decision to introduce an upstream change into RHEL is a team decision and the team is large: developers, quality engineers, support personnel, product owners, and various partners all work together on priority and feasibility.  Once a decision is reached and commitments are made, only then do developers and quality engineers begin writing code.  As you probably know, in the most congenial of rivalries, developers try to write code that nobody can break and quality engineers create batteries of ways to break the code developers write.  This brings us to the second key place where Red Hat invests: tests.

We write tests for everything: unit tests, systemic tests, kernel and userspace ABI conformance tests, performance tests, dependency tests, architecture tests, driver tests, load tests, and many more.  Having tests is foundational, but it is their application that brings meaning.  This brings us to the third key area where Red Hat invests: process infrastructure.

For the last several years, Red Hat has worked on a series of “Always Ready” operating system initiatives.  The goal is as simple as the name suggests: at any moment in time the OS is ready for release.  It’s easier to describe than it is to implement. In complex systems, so many things can have unintended consequences.  To handle this we use layers of automation, incrementally building confidence in changes, before they are integrated and released into the distribution.  Here is a high-level sketch of the process every single change in RHEL must go through to be included:

When a change is targeted at RHEL, multiple incremental steps occur before it is actually included.  Changes are built, but the only certain outcome is that a CI system will run a suite of tests on the builds (the build is not yet made available for general use).  If those tests pass, a second round of preverification specific to the code change occurs (not yet good enough).  And if those tests pass, the change is tentatively included in the errata system and subject to further verification (it’s still not ready to publish).  Systemic test suites run on the combined whole to verify the gestalt functionality.  And if those tests pass, the build will finally make it into the space where CentOS Stream systems recognize it as an available update.  It’s a long pipeline and many changes move through it every single day.  For those interested in more of the vision and architecture of this system, you can read more in CentOS Stream is Continuous Delivery!

While the description of this system may seem elegant and reassuring, watching it in action can feel quite the opposite: The more testing is done the more bugs are found- and Red Hat does a whole lot of testing.  Historically, RHEL development has been done behind closed doors, isolating people from the routine bug identification and remediation process, only allowing the world to see the end result.  In the future, as RHEL development becomes more transparent, as we approach RHEL 9, this process will become uncomfortably visible.  While the testing systems are built to prevent such failures from reaching end users, anybody who wants to look deeper may be surprised at how messy operating system development can be!

Finally, for those who wonder how soon all this will map to CentOS Stream, we have good news: it is already happening today with RHEL 8.4 and CentOS Stream 8!  At the same time these RHEL builds are verified, they are also delivered to CentOS Stream.  Of course we aren’t done yet: CentOS Stream has not yet realized its mission of adding a developer community around RHEL, that is where we are headed, into a place where there are more options to engage with Red Hat and shape future RHEL.  There is always room for improvement, from better tests to more facets in future collaboration, we are excited to share building RHEL with you so that we build a better OS with and for you.

21 thoughts on "How RHEL is Made"

  1. Neil Rieck says:

    I think I can see "the invisible hand of the market" (er, IBM suits)

  2. Vac says:

    Who cares anymore?

    1. rob says:

      my thoughts exactly. You don't break a support commitment and come back from it. RH is dead to me.

  3. Jan says:

    When „continuous delivery“ and permanent change is oh so desirable, why does RHEL itself not follow that scheme? Obviously because there is a strong demand for the point release, security fix only, long-term stable approach.
    No one has a problem with CentOS stream as long as it was a choice you could consciously make for yourself. Both approaches could have coexisted in CentOS, other distros have similar approaches.
    But you decided to force something onto your user base without asking, because RedHat as a company sees benefit from it. And even worse you forced it onto the users when a lot of them were right in the middle of migration, based on the promise, that CentOS 8 would follow the same release model as prior versions FOR 10 YEARS. You decided to pull the rug from under your community when you announced the currrent release model will be abandoned in just a year. People used to be here, because of the long term support and stable release process. If you had just a bit of dignity you would have announced that change for CentOS 9 now and kept stream 8 and CentOS 8 running in parallel. You decided differently for whatever reason.
    For my company >I< decided to stop rolling out CentOS8 immediately. Luckily we just started and it’s only a few dozens systems, yet.
    We will switch to Oracle Linux 8 as a “plug-in replacement” so all the efforts we put into automating, etc is not totally lost. We are an Oracle customer anyway due to our Solaris installments (had them since SUN times). We won’t buy support, but believe me, I am never going to put a dime in RedHat support ever again. In the midterm we will establish Debian in our data centers so we have at least one choice that is truly vendor agnostic.
    We also have quite a large budget run stuff from RedHat’s owner - I will put extra effort in reducing that as much as possible.
    We switched from Informix to PostgreSQL for similar reasons.

    1. commonsense says:

      I'm really impressed with this statement:
      We will switch to Oracle Linux 8 as a “plug-in replacement”

      You don't like Red Hat trying to make Centos something more valuable than a mere Rhel clone "for free" but you are quite happy supporting Oracle, because they have demonstrated to be the most generous IT company in the world and main open source supporters... (ironic mode on)
      Are you cheating ?

      Question: Why do we think we have the right to ask something for free? . Do your customers do the same with you?

      Open source and communities is a symbiotic ecosystem where everybody get benefit from collective work.
      Are you providing some value back to the community?

      Let me explain that Oracle Linux is a RHEL clone which Oracle is using (thanks to Red Hat's openness and investment) to obtain money for free but not investing a cent in open source communities.
      Is this what you want to support ?? If you are sure, ok, go ahead, but I don't think this is the best way.

      Using Centos Stream doesn't look a bad idea and is a way to contribute to Centos. I usually use Fedora, and I know other companies using both Fedora and Centos, so using Centos Stream doesn't look that bad as this is much closer to Rhel than Fedora.

      On the other side, I don't think companies can raise ethic claims or right to receive products they don't pay, specially when when they don't contribute back to communities or and don't provide some kind of free service to society.

      1. DisappointedSteve says:

        Do you use Fedora for your Desktop? Because I can't imagine this comment coming from a system administrator, who's in charge of system stability. Nobody wants to patch and pray for it not to break compatibility, with whatever custom software company runs.

        Switching to OL is not a whim. I don't think any administrator is delighted to do that. It is frustrating last resort option left as drop-in replacement. When you have slew of developers scrutinizing and optimizing code on particular system, you can't just "go away" from it. You plan stable system environment and roll it out on big scale, when respected vendor has promised 10 year support. Then all your hard work is nullified overnight.

        They should better know their audience, Open Source community will never pay if it's forced to do so. It will just switch to the alternative. Maybe in short term they will get miniscule boost in licenses, while companies find and test alternatives, but eventually people will switch distros.

        RedHat knew it very well, but obviously IBM didn't. They just butchered RedHat's respect and image for a "quick buck".

      2. Jan says:

        Commonsense, you are missing the point. It's not a about RedHat changing their approach to CentOS, it's the short notice on which they are cutting the release model. This is what destroys confidence and trust in RedHat.

        Also, we are not talking about old RedHat here anymore, this is IBM's RedHat. Oracle has a bad record when it comes to supporting open source projects? IBM's is not much better.

        And, no, it's not about "free as in free beer". My company hires companies that contribute to the open source community, e.g. PostgreSQL. We have paid for developments that were open sourced. We pay contractors from the open source community to fix bugs for us that will go to the upstream projects. There is many ways to give back. Getting caught up in RH's licensing is not a way I want go. But again, this is about CentOS - not RHEL!

        And when you blame Oracle for just cloning RHEL - you are aware that RedHat uses a lot of stuff in RHEL that others contribute - for free?
        Take RHEL default file system XFS. Contributed by SGI and afaik now mainly maintained by Oracle engineers. Did RedHat pay them for that? No, they just "took it for free".
        Stuff like GlusterFS relies on XFS functionality. So this is not a one way road. You may like RH better than Oracle, but after the IBM acquisition I would doubt they will differ that much soon.

        And no, switching to Oracle Linux is not a love marriage. Right now it seems to be the lesser pain.

        If RedHat had announced to switch the CentOS release model in 10 years or maybe even 5 years and keep Stream and point releases going in parallel for that time, I am quite sure Stream would have gotten more intention that it will probably get now, when people turn their backs on CentOS.

        What they did now is destroying the basis of a lot of people mid and long term planning and that is what people are angry about, and correctly.

  4. > "Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?"

    This comes across a bit tin-eared. Stability doesn't just mean the processes you follow today. It means also being able to rely upon promises about processes that will be followed tomorrow too. That's the question that users of RH-produced things are now asking. When RH tell us that we can use product X, because they're going to support it until 2029, can we rely on it?

    1. Sure, the way people feel and think about the announced change is multi-faceted, but I'm only talking about one facet. This blog post covers the technical aspects of how code changes in RHEL are built and tested. This information reflects directly on what people can expect if they move from CentOS Linux 8 to CentOS Stream 8, by way of operational confidence. Trust issues about future changes in direction are clearly beyond the scope of something I can credibly speak to. At best I can reflect on the last 20 years of the changes in direction Red Hat has made, but I'm just one Red Hatter in many on the score.

  5. Total Loss says:

    I have been using Red Hat for over 22 years, personally and professionally. RHEL6 was, to me, the best version ever, RHEL 7 and beyond is to much. It seems to me that money and pride are your two basic downfalls. You (IBM\Red Hat) don't give thought or care about your base users. I just don't see a continuous stream being a good thing. And even if I'm wrong, there's the fact you make decisions because you want them, not based on what the customer wants. Occams Razor says the simplest solution is probably the best, why don't you listen?

    1. Mike McGrath says:

      If you don't want a continuously updating OS, and don't care to see what's coming in RHEL to make sure we don't break something you need that's fine. If you want an enterprise OS, we have one of those too - the best in the business.

  6. Tecnical says:

    Just NO! This is just excuses.
    No matter what you guys say, the Stream is not a stable release.

    1. That is wher you are just wrong. Stream is just as stable a Debian or Ubuntu.

      When Debain fixes a bug, they roll in a new package. When Ubuntu fixes a bug, they roll in a new package.

      So, when Red Hat fixes a bug in stream, they will roll in a new package, These paackages will not be significantly differnt thanthe current version unless they are a rebase. How many rebases between 8.2 abd 8,3?

  7. Anderson Zardo says:

    Please announce that plan to switch focus to Centos Stream was delayed to Centos 9 and Centos 8 will keep the maintance updates until 2029 as intended. Anything else is bullshit. You guys is still in time to do that.

  8. Hadi says:

    This is not a good because it will only annoy your customers who will not pay for a redhead service anyway they will just go to Rocky Linux or Debian and the good reputation that Red Hat have made with the free and open source community will be destroyed. If people really need professional support they can just buy a Red hat Linux. But Santos is more for people who can’t afford it or don’t need the support. But if this is still a priority then at least delay it till CentOS 9 so we can get the promised 10 years of the board until 2029

  9. Centos says:

    I was used Red Hat before RHEL/Centos exist and start road with Linux. First my system for main machine in my company Was Fedora. Past a few years i see Fedora have too much upgrade and isn't too much stable for my.
    Next system on main machines Was Archlinux and Centos. Archlinux because is simple and very fast, and Centos becouse is very stable. Archlinux was bad idea if you expect high stability. Now I know that only Centos is very stable. I was with us more that 20 years. But I know this that i can't risk with next "CentOS Stream 8". 4 weekes ago i Was migrate form Centos 7 to Centos 8 and now i see that end of Life will be not 10 year but 1 year.
    Probably many people will not trust you in matters of "CentOS Stream 8", since you were able to eliminate Centos within a year, who will guarantee that someone will not get an idea again and "CentOS Stream 8" will not share the fate of CentOS 8 and will not end its existence within a one year.
    I see Linux Red Hat it is ended "Open Source".
    Probably many person migrate to Debian like last good stable system for server machine high critical stable.
    Probably many people will not trust you in matters of "CentOS Stream 8", if you managed to liquidate CentOS within a year, who will guarantee that someone will not get an idea again and "CentOS Stream 8" will not share the fate of CentOS 8 and will not end its existence in during the year?
    People who do not care about RHEL's support will probably not choose anything under the sign of the red hat, because apart from RHEL it will not be associated with stability.
    May someone not find out a few years after the closure of CentOS, it was a mistake with the closure of CentOS because these two solutions could exist simultaneously.
    Once disappointed customers - yes customers, because the benefits were double, despite the fact that CentOS is Open Source. You give us the product without support and we test it and report bugs. After the last update, I reported errors in the operation of the main service myself.
    If you boasted that you have such good testers in RedHat, how could a stable version of Centos come out with a damaged BGP demomen in the form of FRR?
    I reported this to the package maintainers.
    You will get a lot of free "testers", thanks to which Red Hat owes its stability.
    I hope this brutal move will not come sideways for you in the future.
    Regarding the RHEL license, I don't mind paying for something stable but I don't find a licensing option for myself.
    Think whether or not to give people from Centos a gate in the form of a symbolically paid RHEL license without support, which will allow you to gain customers and at the same time get out of this situation without losing total trust in the market for your products.

  10. swa says:

    Then why has my workstation today a bunch of LibreOffice CentOS 8 Stream updates for which not all dependencies are met? Are you saying that dependancy checking is not part of the CI method? Are you converting Stream into a 'in practice there is no point in time you can install the software as dependencies are not guaranteed to be met'?

    1. Hey, thanks for the note. A couple other people filed BZs and we're aware of the problem. The RHEL->CentOS Stream sync tooling will be getting an update soon to prevent this from happening.

      1. Dirk says:

        And thus the destruction of centos as a RELIABLE platform finally had begun 🙁

  11. boxbbcar says:

    Reported the shift to our developers. We have just finished moving everything to Cent7 as dictated by the companies we collaborate with. There has been a steady push, and considerable development to get the requisite product running on windows. This may be the last straw that pushes the product solely to windows. Little matter to me, have trained in both. It will certainly streamline the IT staff.
    The responses I see here defending IBM (Not RedHat, we know who is pushing this)are simply putting lipstick on the pig. The question that first arose when we heard of IBM buying RH was, "how long till they kill of CentOS?" Well, now we have our answer

  12. says:

    With this move, Red Hat has now officially made CentOS users their guinea pigs.

Leave a Reply

Your email address will not be published. Required fields are marked *