CPE - Community Platform Engineering - is the engineering group within Red Hat which does a lot of the behind-the-scenes work that makes the CentOS and Fedora projects possible.

CentOS Stream Project Update 2020-02-28

We would like to welcome you all to our first blog update on the CentOS Stream initiative. Over the course of this initiative, we will share regular updates on our plans, our progress and our deliverables. The CentOS Stream team is currently working within a Scrumban framework broken down into two week blocks. This allows the PO and team to plan and prioritise work for each block allowing stakeholders to gain updates on progress, plans and deliverables each fortnight. Each fortnight, stakeholders will review the block deliverables and provide feedback to be taken into consideration when planning the next block. Ensuring that CentOS Stream is delivered in line with changing requirements and expectations as the project build progresses.

The CentOS Stream team has made significant progress throughout February where their focus was on the cornerstone foundational build phase. Block 1 and 2 (2020-02-03 to 2020-02-21) delivered:

• Nightly composes:
  • Reports are being generated internally for now until a bug fix has been resolved
• CentOS QA can consume composes and the test suite can now run against both Stream and Linux
  • This has also caught three issues that would have made it to downstream otherwise
• The module for Stream this week is PHP:7.2
  • This is ahead of 8.1 in RHEL
• Accounts in git.centos.org are created

Block 3 deliverables were identified and are currently in progress. See below:

• Push to Git process being developed
• Scaffolding up to watch brew tags
• Python:3.8 is expected to be the next module for Stream*
  • This is dependant on a bootstrapping task relating to upstream updates that come from CentOS Linux to Stream
  • Information gathering with package maintainers on package lists around debranding/rebranding for future upstream & downstream use

AAA: FAS replacement project update 2020-02-28

The month of February was a very busy month for the CPE AAA team and community contributors working on this initiative. Great progress was made in the development phase of the AAA: FAS replacement build. Sprint 2 and 3 resulted in the completion of multiple user stories which added user functionality to join groups, change email address and password, disable account, database access along with putting a mapping solution in place for users moving from the current FAS to the new FAS (potential name incoming!). We also came to the end of developing our wireframes and mapping our user experience flow. Unit tests were carried out regarding password controller and the current codebase.

We received great support from the wider CPE team as well as Patrick Uiterwijk to allow us progress with user stories by gaining permissions and merging PR’s for the integration of CentOS CI. Christian Heimes assisted us greatly with sharing his knowledge regarding FREE IPA and answered numerous questions to allow us to move forward.

Sprint 4 began on Thursday the 20th of February. This sprint will focus on development tasks which will include working on FAS Json, Free IPA, API, Fedora Messaging integration, continuous deployment to stage environment, developing a secure coding tool to ensure code adheres to best practice, as well as continuing working on user functionality user stories. Please see our github board here to view current activity.

We also received some sad news since our last update, that we are losing a team member, Rick Elrod, as he moves on to pastures new with the Ansible team. Rick provided an excellent POC for AAA which is leaving us in good shape to continue on as planned. Thanks Rick and we will hopefully still see you around as a contributor going forward. We also welcomed a new team member Leonardo (Leo) Rossetti who joined at the start of Sprint 4 and has already hit the ground running. Leo is currently working on our FAS JSON user stories.

Regarding delivery of AAA, we may look at a phased release , this current phase focus is on the development of AAA to be delivered by 3/31/20. It is looking likely that the deployment of AAA will happen in a later phase due to requiring System Admin assistance. We are likely to gain this on the completion of the Colo Move (which is our planned data center move), approximately in mid April. We are inquiring to see if deploying to staging is possible within this phase to allow for a long testing period. I will provide an update on this in our next blog. The integration of CentOS will be worked on within an additional phase following the completion of AAA centric stories for Fedora.

On a final note, I would like to commend the CPE AAA team on their collaboration and productivity throughout this initiative even in the face of unknowns, team changes, cross team dependencies and other challenges, they continued to proactively work together and find solutions to keep this initiative moving forward.

We welcome all feedback, thoughts and contributions as we progress through this project. Please feel free to comment on any issue to log your thoughts.

• For more information regarding outstanding issues, please see here.
• To view our current scrum board, please see here.

Fedora Data Centre Move Project Update 2020-02-28

Hi Everyone,

As you may or may not be aware, last year Red Hat made the decision to move data centers in 2020.

The lease on the current data center in Phoenix was due to expire in 2020 and Red Hat negotiated a better lease with a provider in Northern Virginia.

This data centre is home to Fedora servers.

So, what does this mean for you as a Fedora user? Very little we hope!

The Community Platform Engineering team have been working closely with Red Hat IT to plan logistics, and other 'fun stuff' to make sure this move is successful and as undisruptive to everyone as possible.

During this planning phase, we identified a need to have a minimum viable fedora offering in place during some key dates to facilitate the move, and allow for the shipment of hardware that is integral to Fedora Infrastructure without halting development - or a whole infrastructure!

Here is the link to the discussion that was sent to the public lists in case you missed it on what a Minimum Viable Fedora would look like: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/PN6RL7XT3V7DVC7MK46H3QDEJPL5FRI6/

The CPE team will be refocusing on this problem to begin technical development of this offering so we are ready to deploy it at the appropriate time.

But for now, here is a very high level view of the Data Centre move outline, and how it will impact you:

The Community Platform Engineering Team will move in two 'waves'

Wave 1

• Week beginning April 13th, the CPE team will move an initial batch of servers from Phoenix, Arizona -> Northern Virginia
• This will not affect the Fedora 32 release aimed for late April.
• However, during the period of 20th March - 3rd of July we would ask you to observe an 'infra-freeze', meaning no new applications deployments and all code changes reviewed by the sysadmin team before deployment.
• During April 20th - May 20th, the Community Platform Team will be working on bringing up a minimal viable Fedora solution for continuity of important services in Fedora for development
• Between May 20th - June 1st, the CPE team will bring the new temporary offering up and redirect services to this instance while the main servers for Fedora are brought down and ready to be shipped.

Wave 2:

• Between June 1st - 15th, The Community Platform Team will ship all remaining hardware from the Phoenix Data Centre to the new one in Washington.
• Between May 20th - July 3rd, Fedora services will run on the Minimum Viable Fedora offering to facilitate a successful move of equipment across country!
• As equipment arrives in the new datacenter, the CPE team will be bringing back services on the new networks and equipment. As with all major moves there will be delays and changes we will only see as we get the services back.
• We are hopefully, and quietly confident, that we will be able to resume Business As Usual (BAU) in Fedora Infrastructure, and have Fedora up and running in early July.

Expected Effects during Move:

There will be a very limited number of builders during this time frame.

• Builds will be slower.
• Composes will be slower.
• Services like koschei will be turned off.
• Services will be 'cramped' with less resources than usual.
• Searches in koji and other tools will be slower.
• Some applications like badges, voting and calendaring may not be available at all.
• Tickets will be slower to resolve. Most CPE engineers will be focusing on rebuilding services in the new center so other requests not involved with that will be put on the backlog.

Disclaimers:

As we move through this project, our dates may change, both for the better and sometimes for the worst so please take the above dates as a *fairly good* estimate for now.

We will be including as many real-time updates on the data center move in our weekly emails to the infra and devel lists.

And while we are planning for as little disruption as possible, there may be downtime during this move so we will endeavor to get ahead of it with messaging out to you all for awareness.

We would finally like to thank you all for your understanding and most of all your patience during the key dates of April 20th - July 3rd so that we can facilitate a successful move.

Please don't hesitate to reach out to us with your questions and we will do our best to answer all the ones we know, and follow up on the ones we don't!