Last week I had a short chat with Brian Stinson and Carl George, from Red Hat's CPE (Community Platform Engineering) team about the work they are doing to enable CentOS Stream. (9 minutes)

Hey everyone,

Back in March we published this blog where the CPE team came to a decision about a future Gitforge. The decision was made to opt for Gitlab as the Forge of choice. We can now announce that the service has been stood up successfully, with Gitlab running this as a SaaS offering on behalf of the community. We are still in the process of making configuration changes and starting to seed content. In the background the CPE team are working through their tooling and configurations to ensure when we launch that the process and experience is seamless for the community.

We expect CentOS Stream distribution repositories to be the first content that will show up in Gitlab. We plan on giving plenty of notice whenever we intend to relocate other things like SIG repos or Downstream source drops from Red Hat. Until then git.centos.org will remain up and running to support your work.

Dear CentOS Enthusiast,

It's been another fairly quiet month in CentOS, but I have a few things to share with you.

News

CPE

Last week I spoke with Aoife Moloney and Stefan Mattejiet of Red Hat's Community Platform Engineering (CPE) group. CPE do a bunch of infrastructure work for Fedora and CentOS, and we've been trying to bring more attention to what they're working on, on our behalf, over the last few months. My conversation with them is on YouTube, and I hope to have more of these interviews over the coming weeks to tell you about specific initiatives.

I would encourage you also to read CPE's weekly updates (October 17th, October 25th), and their Q3 summary for more detail.

If you have suggestions or requests for what CPE should work on in coming sprints, please get in touch with your ideas. You can email the centos-devel mailing list, or contact Rich Bowen directly off-list at rbowen@centosproject.org

CentOS 6 End Of Life, November 30th.

This is your final warning that CentOS 6 will be designated "End Of Life" on November 30th. After that time, it will receive no more updates.

Fedora 33

Our friends over at Fedora released Fedora 33 last week. Celebrate with them by attending their release party this weekend! Register here: https://hopin.to/events/fedora-33-release-party

Updates

A number of people have noticed that there has been no activity on the centos-announce mailing list in October. That's because all of the activity is over on the centos-cr-announce list, as we prepare for a 7.9 point release, which we expect to happen in the coming couple of weeks.

Meanwhile, we encourage you to keep an eye on the Building 7 page for daily updates of the status of that release.

Events

FOSDEM Dojo, February 4th and 5th

Although FOSDEM looks different this year, we plan to continue our tradition of holding a CentOS Dojo on the day (or days) before FOSDEM begins.

This event will take place online. Doing this online gives us access to a wider pool of speakers and attendees - this means YOU! The call for presentations is now open.  The actual schedule will depend on submissions that we receive, but we are tentatively hoping to run the event on February 4th and 5th, ahead of FOSDEM starting on the 6th.

We are looking for presentations about anything that you are doing in, or on, CentOS Linux and CentOS Stream. This includes, but is certainly not limited to, SIG activity, research using CentOS, useful tools/applications that can be deployed on CentOS, or topics around community engagement.

More details about the event will be available on the event wiki page as soon as we know more.

DevConf.cz, DevConf.in

The Call for Presentations for both DevConf.in and DevConf.cz are now open. The .cz CFP closes on November 6th, and the .in CFP closes on November 15th. You can access both of them at cfp.devconf.info.

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

Sent a mail to the devel lists for both Fedora & CentOS with questions
that had answers relating to the topic Accouns & Permissions. It has
been sent to devel-announce at fedoraproject.org &
centos-devel at centos.org. Here is the link to the hackmd doc I used to
write the email before copying it across to my email to send
https://hackmd.io/1pjX1cVnTjekOLVowj5UiQ?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 7.9.2009 pkgs pushed to CR repo, next step is to install tree
and artifacts such as iso & cloud images.
* Working a lot with the AAA/Noggin team on CentOS account integration
and making good progress

CentOS Stream

* There are now cloud image updates available for CentOS Stream
* There are some documentation updates here on unshipped packages &
Stream feel free to read up!
https://wiki.centos.org/FAQ/CentOS8/UnshippedPackages

Fedora

General

* F33 will be released next week!!
* The team got an F33 nightly compose finished too!
* There is a dashboard available to view the performance of Anitya
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/8Zi9LU5Mz/anitya?orgId=1
* And there is capability to do the same for the packager workflow
pipeline which we are still working on

Staging Environment

* Build system nearly done - waiting on a firewall change

Noggin/AAA

* New estimated deployment date for Noggin is 29th January 2021.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 (On Freenode IRC)

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting (On Freenode IRC)

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Hi there,

I'm Aoife Moloney. You may remember me from such communications as the CPE office hours, Data Centre - what it means for you, and The Future of Communishift.

Over the last three months, the Community Platform Engineering team (or CPE for short as it's long to keep typing) have been working on a few projects, and generally surviving 2020 like everyone else. But we made it, and so did our projects! Mostly… 🙂

 

Over the last three months we worked on:

  • The Great Fedora Data Centre Move of 2020
  • Noggin
  • CentOS Stream
  • Packager Workflow Healthcare (Always check with your maintainer before taking this workflow. Side effects may be, but not limited to, frustration, tears, elation at successful builds)
  • Fedora-Messaging Schemas

 

We also had our long standing (and long suffering) ‘sustaining team’ on the front lines who are daily maintaining and running both the Fedora and CentOS infrastructures and responding to issues, bugs, etc. And doing a damn fine job too.

 

And we attended and participated in a few conferences too, namely Nest with Fedora & DevConf US.

 

So, what did we as a team overall achieve in these last few months?

 

CPE Infra & Releng Team 

This team was led by Pingou, and its members in Q3 were Mark O’Brien, Michal Konecky, Fabian Arrotin, David Kirwan, Kevin Fenzi, Vipul Siddharth, Stephen John Smoogen & Tomas Hckra.

This team is a sub team of CPE and focuses on lights on work in both the Fedora and CentOS infrastructures. We will always have some of our team members working in this way each quarter as it is good to have a break from scheduled project workloads and take a foray into the (sometimes) chaotic world of infrastructure maintenance, aka FIRE!!! 🙂

What they did: 

  • Changed their name. May we present: CPE Infra & Releng Team - oooooh, aaaaah!
  • Vipul & David worked with Fabian in the CentOS infra and did something with openshift clusters & migrated the kojihub for https://cbs.centos.org to a new infra
  • Kevin and Smooge moved all of the Fedora infrastructure. 117 servers.  Let that number sink in.
  • Pingou & Michal did a ton of babysitting toddlers 🙂 They moved a lot of scripts over and things are working well
  • Tomas helped bootstrap F33 - oh yeah!
  • And Mark had (not biologically,  but in sentiment) a baby! And became an admin for some of the Fedora Infrastructure.

Why its good:

  • The name change represents what this team works on and is easy to understand instantly. Plus, naming is hard so we wanted to keep it simple 🙂
  • Helped release F33 beta
  • We doubled down on toddlers allowing to build more automation around the infrastructure
  • Assisted with the fedora datacentre move for minimal disruption to the fedorans day to day lives
  • Helped keep CentOS CI operational - and then helped put out the flames when it caught fire 🙂
  • Over 500 tickets across both Fedora & CentOS infra + releng resolved by this team - that is some seriously good firefighting!

 

Fedora Data Centre Move

This dynamic duo was Kevin Fenzi and Stephen Smoogen, with supporting cast members from both CPE and the community along the way. The goal of this project was to successfully move a (large) number of the Fedora infrastructure hardware from one datacentre to the other without too much chaos. And considering the world wide pandemic that happened right at the beginning, they did a pretty fine job succeeding. Some additional services are still being added to the infrastructure in its new home in IAD, so if you notice a few still missing, we are getting to them slowly but surely and thank you again for your patience and understanding during these last few months!

What they did: 

  • Moved a ton of servers across the country of the United States
  • Kept critical services in Fedora Infrastructure alive during the move
  • Worked an uncountable amount of hours!

Why its good:

  • We got some new hardware!
  • The team carried out some resilience testing in the new data centre which means more reliability for the infrastructure should bad things happen 
  • Updated records and warranties were a passive benefit of this move too

 

Noggin

This team was led by Aurelien Bompard, and its members in Q3 were Ryan Lerch, Nils Philippsen & James Richardson. The goal of this project is to replace the current FAS system with a newer one and migrate the CentOS accounts to the one FAS instance (Noggin), which will mean our team has one authentication system to maintain for two infrastructures long term. This team has been working to a November 2020 deadline, but unfortunately during Q3 the team faced a number of challenges such as a delayed staging environment to test in due to the data centre move, then when we got it, realized their plugin they spent time developing was not going to work long term and now have to redo a bit of work in Q4. There were also a lot of holidays and personal events for the team in Q3 because everyone is human and entitled to a life 🙂 They have re-scoped their work for Q4 to make sure what's delivered is sustainable and reliable long term, more people have joined the team including some sys-admin for support along the way, and are now looking at delivering Noggin in full by the end of January 2021. 

What they did: 

  • A lot of ipsilon investigation
  • Added a spam curtailer service to Noggin
  • Added an agreements section for users to select their user preferences 
  • Deployed Noggin to staging but found out the way they did it wont be good for the project long term
  • Had a little cry about developing a plugin unnecessarily, hugged it out and then re-planned dates and the work we need to do in Quarter 4 to be able to deliver a better, more reliable and robust service in January 2021. Queue Noggin’  - Rise of the Phoenix Project

Why its good:

  • We knew where we went wrong, learned a lot both technically and as a project team for it, and were able to call the mistakes out and get the support we need to get the project back on track. Just a little bit later than we wanted.
  • We still created a solution that will meet the needs of both the CentOS and Fedora community users, and once we have the correct configurations in place and are ready to be tested we look forward to your feedback!

CentOS Stream:

This team was led by Brian Stinson, and its members in Q3 were Johnny Hughes, Carl George, Mohan Boddu, Leonardo Rosetti, James Antill & Siteshwar Vashisht

What they did: 

  • A lot of darn package & module building
  • Light hearted threats to their PO to teach her how to convert a CentOS Linux distro to CentOS Stream using the new release package - which she did! 🙂
  • Kept CentOS Stream compose up to date with RHEL nightlies
  • Launched the centos-stream-release package - Big deal. Like, huge.

Why its good:

  • CentOS Stream is continuing to stand on its own and becoming a more robust distro
  • There's lots more content in Stream for its users
  • Users can now swap from CentOS Linux to Stream easily

Packager Workflow Healthcare:

This team was led by Will Woods and its team members were Adam Saleh and Stephen Coady with Pingou in a part-time consulting/reviewing role. The team took a look into the Fedora packager workflow and tried to identify weaker points in the chain, and spot times that are more prone to downtimes.  They are finalizing a report of their findings to send to the community lists with hopefully a ‘next steps’ section that they feel will help reduce the issues packagers face sometimes in Fedora. Its a work in progress, but to have some data to read and understand is a great launching point.

What they did:

  • Refined the monitor-gating script that monitors the packager pipeline to enhance its performance
  • Picked a certain date range and got a database dump to pull metrics from into graphana to chart uptimes of applications within the pipeline
  • Created a diagram of the pipeline to help understand how packages flow through the fedora infra

Why its good:

  • The diagram of the packager workflow process is a great resource for both packagers and new contributors of the fedora community to refer to and help understand how things work
  • The team also have some recommendations they are working through with management and the wider CPE team to identify possible next steps and how we can improve the packager experience long term by adopting better monitoring.

 

Fedora-Messaging Schemas:

This project was also being worked on by the Noggin team part-time, so Aurelien Bompard, Nils Philippsen & Ryan Lerch. We needed to pause this work around the start of September and we hope to be able to return to it over the next quarter - October, November & December.

The guys have a github board here with a cookie-cutter schema available and a list of apps they were working on, so if you want to help out on this one, please feel free to visit the board and grab a card! 🙂 

What they did: 

  • Created a board to track the work being done and whats left to do https://github.com/orgs/fedora-infra/projects/7
  • Created a template schema 
  • Created a list of applications that require a schema update
  • Added some schemas to applications that need them

Why its good:

  • This will help us progress the retirement of fed-msg in 2021
  • It will also give applications, and application maintainers, access to new fedora messaging schemas for more faster & reliable notifications.

 

And that, my dear friends, is Quarter 3 for CPE.

Take care all, and see you around IRC! 🙂

 

Aoife

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-17.
Im gone a little bi-weekly lately with this report. This has been more
circumstantial with our quarter 3 projects ending and launching
quarter 4 work, and will get back to a weekly report now that Q4 is
underway.
So, the updates for both Fedora & CentOS are below, and if you want to
visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
you can then use the header bar on your left to skip to Fedora or
CentOS updates that interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

There is now a blog post out on the Fedora community blog on the AMA,
you can read it here
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/
Im going to be sending one mail per week to the CentOS and Fedora
devel lists on Fridays, dedicated to one topic.
The first topic will be Permission and Access in GitLab. I will pull
the questions and answers from the hackmd doc into an email body to
try to facilitate dedicated discussion around this topic. I would like
to just set expectations now that I don't consider myself to be
technical enough to weigh in on the discussion, but I am looking
forward to reading and learning from the conversation.

Project Updates

*The below updates are pulled directly from our CPE team call we have every week.*

Fedora

General

* Go NoGo meeting was on 15th Oct. F33 was No GO and new date is moved
to 2020-10-27

Staging Environment

* Working on getting OpenQA moved over
* Adding boxes to CentOS to test IPA

Noggin

* After the team reviewed the work still left to do to be able to
deploy a robust and secure solution, we discovered that there is more
development work to do and we are now working towards a release date
of January 29th 2021.
* Accounts & data will remain secure in the current solution until we
can cut over, we have established a monitoring plan to derisk this
extended time the team needs to complete their work.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

CentOS Updates

CentOS

* New CI admin added - mobrien, welcome!
* Plumbing for duffy being worked on currently and will be deploying
to staging in the coming weeks

CentOS Stream

* Looking at ODCS services in the build system
* Also working on deploying mbbox operator to the system too
* centos-stream-release package is also out, go check it out!

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

Dear CentOS enthusiasts,

The past month has been quieter than recent months, in terms of news, and I was very occupied with some other things last week, so I hope you can forgive the delay in getting this month's newsletter out.

If you want to receive notifications of new newsletters in future months, please subscribe to the centos-newsletter mailing list to be the first to hear of new editions.

News:

Embroidered shirts and sweatshirts at HelloTux

You can now purchase embroidered CentOS tshirts, polo shirts, and sweatshirts, in the store at HelloTux. HelloTux also produces garments featuring our friends Debian, Ubuntu, openSUSE, and various other Linux and open source projects and brands.

An article with a little more detail about the process may be found on the CentOS blog.

Board welcomes Brian Exelbierd as director

On 2020-07-08 the CentOS Board of Directors welcomed Brian "bex" Exelbierd as a new member Director and Red Hat Liaison. This, and other board news may be found in the minutes from that board meeting.

Creation of Infrastructure SIG

In the September board meeting, the Board approved the creation of the Infrastructure SIG, which will provide a place for coordination of all planning and decisions which effect the infrastructure that enables the CentOS project. Aoife Moloney was named as the initial chair of this group. More details about the function of this SIG may be found in the article announcing the creation.

Updates:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during September:

 

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-09-20.

I found that if you want to skip to the hackmd, you can use the view
link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and then use the
header bar on your left to skip to either the Fedora or CentOS
updates, whichever interest you.

I'll also be adjusting these updates in the coming weeks to make them
a bit more direct to consume. Thanks for giving me this feedback in
the CPE survey, I want to deliver value to you all, so it's great to
KNOW what you find valuable first hand

General Project Updates

As a reminder, below are the projects the CPE team are working on for
the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

We have recently held our Q4 planning session and the CPE review team,
Fedora, CentOS and RHEL BU have voted the following projects for
action in Q4, which is the months of October November & December:

* OSBS for aarch64
* Fedora-messaging schemas

We are continuing to work on CentOS Stream and Noggin and took these
projects as confirmed when looking at what other work our team could
realistically complete in the Q4 period, given that there's both
Thanksgiving and Christmas time off to consider, plus any time off our
team wishes to take.

The taiga cards of Noggin, CentOS Stream, OSBS for aarch64 and
fedora-messaging schemas will be updated next week with what our team
hopes to deliver in the next quarter on each of the projects.
Our project board is here (it's just not updated properly - yet)
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

Misc

GitLab

Thank you so much to everyone for adding your questions to the doc for
the GitLab AMA session on Thursday 10th September, and for your
attendance on the day during the call.
Here is the full AMA transcript
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_with_gitlab.2020-09-10-13.31.log.html
however it is a bit confusing to read so we got a few great
suggestions to have dedicated topics like Message Bus and Branching,
etc go out to the devel lists to discuss. I'm happy to start this next
week, but I will collect the questions related to each topic and
propose a cadence to send them out first to discuss, so people dont
miss mails and know the week ending 2nd October will be (for example)
the topic of Group Permissions - What do you think?

GitLab have also agreed to answer the questions, we have asked them to
do so within 2 weeks of the AMA so as soon as this is complete I will
let you know so you can read through them on the hackmd link.
The link is here where we asked you to contribute your questions and I
will be posting answers once we have them underneath
https://hackmd.io/RW8HahOeR7OJPON1dwuo3w

I really appreciate your involvement with this as we begin to dig
deeper into how this might play out next year and what way it should
for everyone's benefit.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Deployed new 4.5.9 openshift cluster for Stream
* The team provisioned EC2 infra for team responsible for
registry.centos.org (we don’t maintain it, so just providing infra,
like Fedora does for Copr)
* They also migrated a bunch of nodes to the new Ansible CI inventory

CentOS Stream

* Using Openshift cluster for engineering work and will be using it to
deploy & test mbbox in our infra
* Scoping and refining work for October November & December

Fedora

General

* 6 of 8 Beta-blockers have fixes for F33 beta
* New release of fedscm_admin
* FMW mac and windows binaries are signed

Staging Environment

* About 70% done installing vm’s (27 left out of 88)
* Still need to bring up aarch64/armv7/ppc64le builders
* Databases need syncing

AAA Replacement

* The team are working on testing Ipsilon in Staging and adding OpenID
Connect Capability
* they are also testing fas2ipa migration script in tiny-stage and improve it
* Add Noggin to tiny-stage environment and test
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is on hold until Noggin completes.
* It will be resumed around December timeframe and is part of our Q4
workload to complete
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been working on more improvements and fixes to the
monitor-gating
* These improvements have led to
* Finding a bug in our testing script
* Improved log messages
* We actually caught a problem!
* The data the team have been reviewing have been from April - July
and have already discovered that so far it looks like Pagure, koji and
bodhi work well
* We see some intermittent problems, but nothing too big, mostly
only spikes in runtime
* Fedora CI still looks like a POC, but functional
* Our test-script hitting timeouts/failing 10% of the time
* Gating (greenwave/resultsdb/waiverdb) looks functional, but
relies on CI and doesn't have as much packages going through the
workflow
* A more formal report will be published soon as part of the project
deliverable so keep an eye on their work!
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Team Info

Changes to CPE Product Owner Office Hours

Following the feedback received in the CPE survey, I will be reducing
my IRC office hours to once per month.

#fedora-meeting-1

* Next Meeting: 2020-10-15 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-10-13 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

Hi everyone,

Welcome to September! Below are the most recent Community Platform
Engineering project updates, and if you want to know more about our
team, see our wiki page here for more information on who our team is:
https://docs.fedoraproject.org/en-US/cpe/

Here are some upcoming IRC meetings:

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-09-03

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-09-01

GitLab AMA Session

* September 10th @ 1330 UTC on #fedora-meeting-1

Below are the project & community updates this week:

GitLab

There will be an IRC based AMA session with GitLab on Thursday 10th
September @ 1330 UTC in place of the CPE PO office hours.
We are still talking to GitLab but we are deliberately taking our time
to make sure all of the technical blockers can be met and the move
will be worth it in the end.
There is very little to no updates in the tracker, but I will include
it nonetheless https://gitlab.com/gitlab-org/gitlab/-/issues/217350
I will also be sending a separate email on details of the AMA session
later this week, such as how to submit questions in advance so there
is content ready on the day.

Note: Since I'm posting this to the blog after the meeting, you may wish to read the

CentOS Updates

CentOS

* Updated ocp.stg to OCP v4.5.6.
* Added a number of users to the jump.ci host.
* Adding monitoring/alerting for NFS slowness to the ocp cluster.

CentOS Stream

* Module push tweaks.
* Exploring how to enable fedora messaging in Stream
* Reviewing documentation on contributor policies before publishing
them later this quarter.

Fedora Updates

Staging Environment

* Services will begin to be deployed this week
* Please be patient as some services will inevitably not work due to
networking errors that the team don't know until they deploy
* Thank you again for your patience and understanding during these
last few months!

AAA Replacement

* Deployment to staging for testing is delayed due to missing firewalls in IAD2
* This has just recently been unblocked so the team will begin some
deployment and testing of Noggin this week
* Wider community testing will be available, estimated next week
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* List of applications that require messaging schemas can be found
here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been reviewing data on how packages are built in the
fedora infrastructure for the last 8 weeks and have gathered enough
information to create a report on their findings.
* This report is currently in draft format, and is going to be
reviewed by the team first, and then sent to the devel and infra lists
in the next 2 weeks est.
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Here is a reminder of what our team has committed to work on in this
quarter of the year:

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

As any open source project grows and matures, the people who have always done all the things can’t do everything any more. CentOS is at that point (really, we have been for a long time) and we’ve been struggling with those kind of growing pains for some years.

Recently, we made a small change in process, so that SIG leads can manage their own SIG membership and permissions associated with SIG membership. This removed some of the load on the people who have traditionally handled this on behalf of the SIGs. This change was made possible by updates in the authentication tooling, and has the Board looking at other areas where we can move more operational things out of the Board, which can then focus on governance and executive tasks.

Another area where this need for maturity has been clear for a while is our infrastructure, where a small number of people handle all of the infrastructure tasks. As our infrastructure grows (in both size and complexity) this has put us in situations where work was blocked because those people didn’t have time to get tasks done. And, sometimes, people like to take a holiday.

This week, the Board of Directors approved a plan to move infrastructure oversight into a SIG structure, so that different parts of the infra can be delegated to a larger group of people. We have already been in partnership with CPE (Red Hat’s Community Platform Engineering team) to do some of these tasks, and this will give greater liberty to spread the load around even further, but also greater transparency around what it takes to be granted that access.

We are still in the process of figuring out what is in scope, and collecting a community of interested volunteers willing to do the work. The CentOS infrastructure is very broad, including build machines, the mirror network, systems with donated hosting, and so on, and we want to be sure that we do this in a way that doesn’t interfere with the work that’s already being done by so many people, while at the same time working to identify more places where volunteers can make a difference.

To be clear, giving administrative access to any systems still requires a great deal of trust and assurance of competence. We’re not giving the keys away to anyone who asks. But people who have demonstrated that competence and trustworthiness will be able to do some of the tasks that have, thus far, been handled by 2 or 3 people. We’ll be working to develop and publish some clear guidelines around who can be trusted with this access, and how to demonstrate that competence.

The exact details of how this SIG will operate are still being worked out. But the Board has approved Aoife Moloney as the first Infrastructure SIG chair, since she has already been sending infrastructure updates to the centos-devel mailing list roughly weekly for some months now, and is a logical person to take on this organizational responsibility. So, thank you, and welcome, to Aoife, for her willingness to tackle this job.
We welcome participation and comment from anyone with the skills or interest to be helpful, particularly if you are already administering any of the infrastructure that the CentOS project relies on.