Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* CentOS Dojo @ FOSDEM is on right now! Links to talks from Thursday
are on the CentOS youtube channel and Rich is playing a blinder
getting all the content uploaded in record time
https://www.youtube.com/TheCentOSProject
* NOTE: 'playing a blinder' means doing an excellent job for
anyone unfamiliar with the term
* Fedora has a booth as well @ FOSDEM this weekend! Make sure you stop
by and say hi to all those great Fedorans who will be manning it this
weekend https://chat.fosdem.org/#/room/#fedora-stand:fosdem.org

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CI team members are migrating Fedora-Infra and Fedora-apps namespace
whcih is one of the last few before we shut down legacy cluster
* There is also an investigation spike on Zabbix upgrade to current
LTS version which will then be rolled-out on the CentOS Infra once
complete

CentOS Stream

* Python39 built and ready to compose
* Dist-git repos are regularly up to date
* Repos are populated in the CentOS Stream GitLab instance and will be
publically viewable in the coming weeks
* Very detailed talks on CentOS Stream given by Brian Stinson & Brian
'Bex' Exelbierd are watchable now on the CentOS YouTube channel -
check them out!

Fedora

* Infra team are assisting with the testing of ipa/noggin for
otp/other cases in stg
* Their also doing a cleanup of a bunch of broken links on koji volume
* Mass rebuild of rpms is done, modules are underway
* FTBFS for the mass rebuild are filled

CPE ARC TEAM

(Community Platform Engineering Advanced Reconnaissance Team....Team)
We have a new sub team in our team, led by Pingou, who are running
advance investigations on some of the tech debt and bigger initiatives
that the CPE team have in our backlog and they have been tackling
Datanomer/Datagrepper tech debt first.
The team have been partitioning the ‘messages’ table of datagrepper's
DB, & hope to be able to test this setup next week
* prod like in openshift
https://datagrepper-monitor-dashboard.app.os.fedoraproject.org
* prod like with a default delta of 3 days
http://datagrepper.arc.fedorainfracloud.org/datagrepper/
* partitioned table + default delta of 3 days
http://datagrepper-test.arc.fedorainfracloud.org/datagrepper/
* using the timescale postgresql plugin [not implemented yet]
http://datagrepper-timescale.arc.fedorainfracloud.org

Noggin/AAA

* We faced some issues with IPA limits and tuning, and 2FA & still
trying to figure out the best way to enforce 2FA with sudo.
* We are getting closer to migrating from stg to prod and once the
Fedora migration is complete, the CentOS accounts will be then
imported.
* NOTE: If you have an account in both CentOS & Fedora and have
different email addresses associated with each, please update your
preferred email address in your profile and look out for an email next
week on your options.
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* Elections pr reviewed https://pagure.io/elections/pull-request/90
* Next is Greenwave & waiverdb
* Board the issues are tracked on are here
https://github.com/orgs/fedora-infra/projects/7

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

Last week we held our traditional annual CentOS Dojo at FOSDEM. We had 216 people registered, of whom 164 (75.9%) actually showed up to attend some part of it. A big thank you to those that turned up and made it a successful event.

In case you missed it, or some part of it, all of the content is now on YouTube.

On Thursday we had four presentations:

  • The Board of Directors had an "ask me anything" session, where questions were fielded from attendees. [Video]
  • Brian Exelbierd and Brian Stinson talked about CentOS Stream. [Video, Slides]
  • Tomas Tomecek talked about the contribution workflow of CentOS Stream, and how that is the process to land changes in RHEL. [Video, Slides]
  • David Duncan talked about building elastic configurations with EC2-Hibernate [Video, Slides]

And on Friday, we had four more:

  • Javier Peña and Alfredo Moralejo Alonso talked about how OpenStack became boring (and successful) [Video, Slides]
  • Davide Cavalca gave an introduction to the new Hyperscale SIG [Video, Slides]
  • Matthew Almond talked about speeding up DNF/RPM using copy on write [Video, Slides]
  • David Duncan talked about building an image pipeline with CentOS Stream and Image Builder [Video]

It was great to get together with the community, even though it was online. We had some great impromptu discussions in the "hallway track", and it was good to see some faces.

We want to do these at least quarterly for the remainder of this year - watch Twitter and the mailing lists for announcements of dates for the next event! We would also like to hear from you what content you would like to see at upcoming events, especially if you'd like to give a presentation about what you're working on.

Hi Everyone,

New Year, same CPE weekly(ish)

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

General Project Updates

We are kicking off Q1 this year with some familiar project faces,
namely Noggin, the replacement of the current FAS system and
continuing our development of CentOS Stream.

Most of our initiatives live here
https://pagure.io/cpe/initiatives-proposal and you can use the new
issue button to submit your own proposal.

Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.

We also have updated our docs section on the initiative process we
follow as we cannot accept everything so please do check it out if you
want to understand our process more
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

GitLab

Being very honest, I've found myself a little bit strapped for time to
give this project its due diligence over the last few months, but
please bear with us/me and expect a more concentrated effort on this
coming into Q2 (April, May, June) of this year. I apologise for the
time a resolution is taking and I really do appreciate all of your
patience.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Community newsletter can be read here
https://blog.centos.org/2021/01/centos-community-newsletter-january-2020-2101/

CentOS Stream

* Continuing to work on Stream 8 pushes and builds
* Investigating how to automate some module pushes
* Reviewing documentation that is available on Stream currently to
identify gaps and where needs improvement

Fedora

* OSBS is building for aarm64 & x86_64 in production since December!
* All of the projects under the fedora-infra and releng namespaces on
pagure have had their default branch migrated from “master” to “main”.
* F34 mass rebuild due to start next week

Noggin/AAA

* New sprint started focusing on testing correct access has been given
per user/account
* Last remaining apps being configured & tested with fasjson API
* Work will be tracked here https://github.com/fedora-infra/aaa-tracker/issues/4
* Our open issues board can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* We are working through supybot and greenwave applications currently
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

## Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

I usually include the below report in the monthly newsletter, and overlooked it this month. So, without further ado, here are the CentOS 7 updates that were pushed out in December:

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during December:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during December:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during December:

Other releases

The following releases also happened during December:

Dear CentOS Community,

As we enter the new year, I'm sure there's really only one thing on your mind, and so we'll start there.

As you are no doubt aware, the CentOS project has shifted focus from CentOS Linux - the RHEL rebuild - to CentOS Stream - the continuously delivered distribution that reflects what will be delivered in the next release of Red Hat Enterprise Linux (RHEL).

Many, many articles have been written about this, and I want to take an opportunity to call out some of the better ones, to help you understand what's happening, and where we go from here.

To those who claim that CentOS Stream will be somehow unstable, I would encourage you to read Brendan's article about how RHEL is made. Things that go into RHEL are not bleeding edge or continually shifting sands. They are small incremental changes which have been baked for a long time.

To those objecting to the term "rolling release", see Stef's article about continuous delivery, and how CentOS Linux and CentOS Stream related to RHEL.

And to those who are pre-judging CentOS Stream without the benefit of even trying it, you should read Jack's article about not knocking it until you try it. (Jack's an Ubuntu fan, but makes a lot of good points.)

Karsten has written an article about the various things that are kept in balance around the CentOS project, and some of the history that led to where we are.

Finally, Scott's article about ... well, all of it ... is definitely worth your time if you want to have a deeper understanding about why people are angry, and why they are right, and wrong, to be angry.

For those of you who are planning to move to Rocky, CloudLinux, or one of the other projects that has sprung up to take the place of CentOS Linux, we wish you - and these projects - all the best. But we caution you to understand that building an OS is a big project, and it's going to take a while for them to get where they're going. Please plan your migration accordingly.

There are other things happening in the CentOS community, but we understand that this one is pretty overshadowing right now.

Hyperscale SIG proposed

A group of developers has proposed a Hyperscale SIG, which will be voted on in Wednesday's board meeting. They propose to focus on solutions around large-scale infrastructures, such as those at organizations such as Facebook and Twitter.

If you are interested in this kind of SIG, and particularly if you are running a hyperscale infrastructure, we welcome your comments and participation.

CentOS Linux 8 (20-11) released

The fourth release of CentOS 8 is now available, as of December 7th. This release is labelled 8.2011 (ie, November 2020) and is based on the 8.3 release of RHEL.

Q1 CPE Priorities

In Q1, CPE will be working on the following priorities:

  • CentOS Stream
  • Noggin/AAA replacement
  • Fedora-Messaging Schemas 
  • Flatpak indexer code merge
  • Debuginfo-d
  • Datanomer & Datagrepper V.2

We'll be updating the centos-devel list as progress is made on these projects.

Happy New Year

We wish you a 2021 that is happy and productive, and hope to see you in person before the year is out. Thanks, as always, for being part of our community.

 

These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.

I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.

Read More

This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time.  CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified.  Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?  If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built.  Let’s get into it!

Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto.  Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch.  Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago.  While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.  

Read More

On 2020-11-11 the CentOS Board of Directors met to discuss ongoing business.

First, the board would like to thanks everybody involved in CentOS Linux 7.9 release.

The Board was in an Executive session, where Red Hat CTO, Chris Wright joined to present Red Hat plan around CentOS Linux and CentOS Stream. A Board discussion followed.

Following up the discussion around the different users' communities impacted by proposed changes, Chris Wright, mentioned to the Board that Red Hat is also reshaping and expanding the RHEL Developer program. The details will be communicated through standard Red Hat channels.

The following resolutions were approved by the majority of the Board :

  • CentOS Stream 8 will continue with contributions for the full-support phase of RHEL 8. APPROVED
  • CentOS Stream 9 will start on schedule with the RHEL 9 Beta. APPROVED
  • CentOS Linux 9 will not start. APPROVED
  • CentOS Linux 8 ends in December 2021. APPROVED

An announcement and detailed FAQ will be prepared in next weeks.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

On 2020-09-09 the CentOS Board of Directors met to discuss ongoing business. The Board discussed the creation of the infrastructure SIG to streamline and foster contributions in this challenging area. 

The creation of the Infrastructure SIG was approved and Aoife Moloney will be the new chair in charge to gather requirements from all actors and define the SIG baseline contribution model. Board members insisted that administrative accesses to the CentOS infrastructure will need to be carefully granted and on the principle of least privilege. Rich Bowen kindly drafted a blog post that goes in deeper details for the next steps and the challenges ahead.

A lengthy exchange happened around the feedback loop from SIG to the CentOS Board. The board would like to invite SIG chairs (or their representative) for discussion when needed, but also hear from them on a regular basis. A communication will follow-up when the board agrees on the best format and frequency for these sessions.

Ralph Angenendt announced he decided to step down from the CentOS Board of Directors. The Board would like to thank Ralph for all his service and hard work over the years.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already 
as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Fedora, CentOS Stream and RHEL delivery

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL