CentOS Hyperscale SIG Quarterly Report for 2023Q1

This report covers work that happened between January 9th 2023 and April 3rd 2023. For previous work, see the 2022Q3 report.

Purpose

The Hyperscale SIG focuses on enabling CentOS Stream deployment on large-scale infrastructures and facilitating collaboration on packages and tooling.

Membership update

Since the last update, the SIG gained four new members (Dalton Miner, Oleg Obleukhov, Vadim Fedorenko and Alexander Bulimov).

We welcome anybody that’s interested and willing to do work within the scope of the SIG to join and contribute. See the membership section on the wiki for the current members list and how to join.

Releases and Packages

Unless otherwise specified, packages are available in our main repository, which can be enabled with dnf install centos-hyperscale-release. Please report any issues with these packages on our package-bugs tracker.

Documentation

We have continued streamlining our user documentation website and keeping its content up to date.

As previously mentioned, we would very much welcome any feedback and contributions you might have for this documentation.

systemd

We’ve enhanced our Hyperscale systemd documentation with instructions on how to test the new builds with stock CentOS Stream. In particular, it now includes instructions on how to debug and update the SELinux policy.

We updated our backport of systemd to v252. Along the way, we further reduced the number of deviations from the Fedora rawhide spec of systemd. We also fixed the accompanying selinux module, and updated the release docs to describe the latest release changes.

Kernel

The latest version in the Hyperscale SIG is kernel-5.14.0-76.hs2.hsx for CentOS Stream 8 and kernel-5.14.0-258.hs3.hsx.el9 for CentOS Stream 9. We are working on a rebase to the latest CentOS Stream 9 kernel and incorporate support for building it for CentOS Stream 8 too.

Container images

Our container build pipeline is fully automated, and container images are built on the CentOS OpenShift CI/CD infrastructure and published weekly on Quay.

We provide both CentOS Stream 8 and CentOS Stream 9 variants at quay.io/centoshyperscale/centos.

Spin images

We have worked with CPE to enable the ability to use KIWI to build operating system images through CBS. This is now enabled in CBS for Hyperscale. During efforts to produce images through CBS, we discovered Koji doesn’t import environment groups. We have worked around this issue in our image descriptions while we wait for this to be fixed. Currently, we are blocked on an issue whereby Koji fails the build due to not knowing all the artifacts to archive.

The remaining task is to figure out how the release pipeline is supposed to work from CBS to the mirrors.

Package updates

We have completed the rebuild of the remaining packages that still had a “CentOS” vendor tag; from now on, all packages coming from Hyperscale will have a “CentOS Hyperscale SIG” vendor tag. This both makes provenance clear, and allows one to set allow_vendor_change=False in DNF to avoid SIG packages being accidentally replaced by stock CentOS packages with higher NEVRAs (which can someone happen when the SIG package hasn’t caught up yet).

We have published new backports of squashfs-tools 4.5.1 and valgrind 3.20.0 for CentOS Stream 8 and CentOS Stream 9. Our existing grep backport has also been updated to 3.9 on both platforms. Finally, we have released a backport of libslirp 4.4.0 for CentOS Stream 8.

The kpatch build in Hyperscale has been updated to 0.9.8, integrating a number of upstream improvements and improved support for clang PGO optimizations.

On the development front, our CI pipeline to detect package updates in upstream CentOS that would supersede our versions and alert us has received a number of reliability improvements and bug fixes, and is now tracking CentOS Stream 9 as well in parallel with CentOS Stream 8.

Intel repository

We maintain a “hyperscale-intel” repository with optimized versions of packages. As part of this work, we have released an optimized version of the zlib package for both CentOS Stream 8 and CentOS Stream 9 yielding significant performance gains. The optimized zlib package is identical to the CentOS Stream zlib package, but with two alternative hash implementations to provide a fast and high-quality hash function on systems supporting SSE4.2 and AVX2 instructions.

DNF/RPM stack with CoW support

The Copy-on-Write stack was rebuilt on top of the latest packaging changes in upstream CentOS.

We’re still in the process of reworking the CoW patchset to address the latest upstream feedback; once the discussion has settled we will publish an updated version.

OpenShift migration

The SIG has migrated its projects and data over to the new cluster.

Health and Activity

The SIG continues to maintain a healthy development pace.

Meetings

The SIG holds regular bi-weekly meetings on Wednesdays at 16:00 UTC. Meetings are logged and the minutes for past meetings are available.

The SIG uses the #centos-hyperscale IRC channel for ad-hoc communication and work coordination; this channel is also bridged on Matrix in the #centos-hyperscale:fedoraproject.org
room. For async discussions and announcements we generally use the centos-devel mailing list. The SIG also holds open monthly video conference sessions to promote collaboration and social interaction.

Conference talks

Ali Koroglu, Davide Cavalca, Daan De Meyer, Neal Gompa, Quentin Deslandes and Richard Phibel attended CentOS Connect and FOSDEM 2023 in Brussels. At Connect, Davide presented the usual update on SIG activities and Michel Salim (joining remotely) talked about ebranch.

Anita Zhang, Davide and Michel later attended SCaLE 20x in Pasadena in March. Together with Alvaro Leiva Geisse, Anita presented a workshop on systemd and its usage. Michel also presented a talk on how Meta collaborates with upstream projects in the Linux distributions space.

In May, a number of SIG members are planning to attend LSFMMBPF and OSS NA in Vancouver. We have also submitted talks for and are planning to attend DevConf.CZ in Brno in June, Flock in Cork and DevConf.US in Boston in August.

As a reminder, we have a page keeping track of our conference presentations with links to recordings and slides where available.

Face-to-face meetup

The day before Connect, we also hosted our second face-to-face Hyperscale SIG meetup. Davide, Daan, Neal, Quentin and Richard participated in person, joined by Alasdair Kergon, Amy Marrich, Jonathan Wright, Fabian Arrotin, Pablo Greco and Shaun McCance. While this was primarily meant as an in-person event, we also had Anita and Michel joining remotely over a conference bridge.

Like the previous one, this event gave us an opportunity to socialize face to face, leading to plenty of great conversation and significant progress on several work items. The meetup wasn’t recorded, but there are notes of the discussion if you’re interested.

We will likely organize another in-person SIG meetup around one or more conference in the coming months, stay tuned for an announcement in the near future.

Live streams

The SIG periodically does work live on Twitch from its official Twitch channel. Interested parties who want to watch and interact with us as we do work should follow us on Twitch to get notified for when we stream.

Planned work

The SIG tracks pending work as issues on our Pagure repository. Notable projects currently in flight include:

• using CBS to build our spin images
• shipping an updated QEMU package in EPEL
• integrate btrfs transactional updates as an optional feature

Issues for the Board

We have no issues to bring to the board’s attention at this time.