Blog.CentOS.org

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* CentOS Dojo @ FOSDEM is on right now! Links to talks from Thursday
are on the CentOS youtube channel and Rich is playing a blinder
getting all the content uploaded in record time
https://www.youtube.com/TheCentOSProject
* NOTE: 'playing a blinder' means doing an excellent job for
anyone unfamiliar with the term
* Fedora has a booth as well @ FOSDEM this weekend! Make sure you stop
by and say hi to all those great Fedorans who will be manning it this
weekend https://chat.fosdem.org/#/room/#fedora-stand:fosdem.org

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CI team members are migrating Fedora-Infra and Fedora-apps namespace
whcih is one of the last few before we shut down legacy cluster
* There is also an investigation spike on Zabbix upgrade to current
LTS version which will then be rolled-out on the CentOS Infra once
complete

CentOS Stream

* Python39 built and ready to compose
* Dist-git repos are regularly up to date
* Repos are populated in the CentOS Stream GitLab instance and will be
publically viewable in the coming weeks
* Very detailed talks on CentOS Stream given by Brian Stinson & Brian
'Bex' Exelbierd are watchable now on the CentOS YouTube channel -
check them out!

Fedora

* Infra team are assisting with the testing of ipa/noggin for
otp/other cases in stg
* Their also doing a cleanup of a bunch of broken links on koji volume
* Mass rebuild of rpms is done, modules are underway
* FTBFS for the mass rebuild are filled

CPE ARC TEAM

(Community Platform Engineering Advanced Reconnaissance Team....Team)
We have a new sub team in our team, led by Pingou, who are running
advance investigations on some of the tech debt and bigger initiatives
that the CPE team have in our backlog and they have been tackling
Datanomer/Datagrepper tech debt first.
The team have been partitioning the ‘messages’ table of datagrepper's
DB, & hope to be able to test this setup next week
* prod like in openshift
https://datagrepper-monitor-dashboard.app.os.fedoraproject.org
* prod like with a default delta of 3 days
http://datagrepper.arc.fedorainfracloud.org/datagrepper/
* partitioned table + default delta of 3 days
http://datagrepper-test.arc.fedorainfracloud.org/datagrepper/
* using the timescale postgresql plugin [not implemented yet]
http://datagrepper-timescale.arc.fedorainfracloud.org

Noggin/AAA

* We faced some issues with IPA limits and tuning, and 2FA & still
trying to figure out the best way to enforce 2FA with sudo.
* We are getting closer to migrating from stg to prod and once the
Fedora migration is complete, the CentOS accounts will be then
imported.
* NOTE: If you have an account in both CentOS & Fedora and have
different email addresses associated with each, please update your
preferred email address in your profile and look out for an email next
week on your options.
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* Elections pr reviewed https://pagure.io/elections/pull-request/90
* Next is Greenwave & waiverdb
* Board the issues are tracked on are here
https://github.com/orgs/fedora-infra/projects/7

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Last week we held our traditional annual CentOS Dojo at FOSDEM. We had 216 people registered, of whom 164 (75.9%) actually showed up to attend some part of it. A big thank you to those that turned up and made it a successful event.

In case you missed it, or some part of it, all of the content is now on YouTube.

On Thursday we had four presentations:

• The Board of Directors had an "ask me anything" session, where questions were fielded from attendees. [Video]
• Brian Exelbierd and Brian Stinson talked about CentOS Stream. [Video, Slides]
• Tomas Tomecek talked about the contribution workflow of CentOS Stream, and how that is the process to land changes in RHEL. [Video, Slides]
• David Duncan talked about building elastic configurations with EC2-Hibernate [Video, Slides]

And on Friday, we had four more:

• Javier Peña and Alfredo Moralejo Alonso talked about how OpenStack became boring (and successful) [Video, Slides]
• Davide Cavalca gave an introduction to the new Hyperscale SIG [Video, Slides]
• Matthew Almond talked about speeding up DNF/RPM using copy on write [Video, Slides]
• David Duncan talked about building an image pipeline with CentOS Stream and Image Builder [Video]

It was great to get together with the community, even though it was online. We had some great impromptu discussions in the "hallway track", and it was good to see some faces.

We want to do these at least quarterly for the remainder of this year - watch Twitter and the mailing lists for announcements of dates for the next event! We would also like to hear from you what content you would like to see at upcoming events, especially if you'd like to give a presentation about what you're working on.

Hi Everyone,

New Year, same CPE weekly(ish)

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

General Project Updates

We are kicking off Q1 this year with some familiar project faces,
namely Noggin, the replacement of the current FAS system and
continuing our development of CentOS Stream.

Most of our initiatives live here
https://pagure.io/cpe/initiatives-proposal and you can use the new
issue button to submit your own proposal.

Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.

We also have updated our docs section on the initiative process we
follow as we cannot accept everything so please do check it out if you
want to understand our process more
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

GitLab

Being very honest, I've found myself a little bit strapped for time to
give this project its due diligence over the last few months, but
please bear with us/me and expect a more concentrated effort on this
coming into Q2 (April, May, June) of this year. I apologise for the
time a resolution is taking and I really do appreciate all of your
patience.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Community newsletter can be read here
https://blog.centos.org/2021/01/centos-community-newsletter-january-2020-2101/

CentOS Stream

* Continuing to work on Stream 8 pushes and builds
* Investigating how to automate some module pushes
* Reviewing documentation that is available on Stream currently to
identify gaps and where needs improvement

Fedora

* OSBS is building for aarm64 & x86_64 in production since December!
* All of the projects under the fedora-infra and releng namespaces on
pagure have had their default branch migrated from “master” to “main”.
* F34 mass rebuild due to start next week

Noggin/AAA

* New sprint started focusing on testing correct access has been given
per user/account
* Last remaining apps being configured & tested with fasjson API
* Work will be tracked here https://github.com/fedora-infra/aaa-tracker/issues/4
* Our open issues board can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* We are working through supybot and greenwave applications currently
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

## Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL