This report covers work that happened between April 4th 2023 and August 31st 2023. For previous work, see the 2023Q1 report.
Purpose
The Hyperscale SIG focuses on enabling CentOS Stream deployment on large-scale infrastructures and facilitating collaboration on packages and tooling.
Membership update
Since the last update, the SIG gained four new members (Raymond Colebaugh, Jonathan Wright, Roberto Campesato and Neil Hanlon).
We welcome anybody that’s interested and willing to do work within the scope of the SIG to join and contribute. See the membership section on the wiki for the current members list and how to join.
Releases and packages
Unless otherwise specified, packages are available in our main repository, which can be enabled with dnf install centos-hyperscale-release. Please report any issues with these packages on our package-bugs tracker.
Documentation
We have continued streamlining our user documentation website and keeping its content up to date.
As previously mentioned, we would very much welcome any feedback and contributions you might have for this documentation.
systemd
This quarter, we revamped the CI/CD process for the Hyperscale systemd backport. The previous implementation was based on Openshift and turned out to be quite hard to debug when something went wrong. For the new implementation, we’re using Gitlab pipelines instead. We moved the systemd-releng repository to Gitlab and set up Gitlab CI pipelines to do daily builds of the latest systemd upstream sources against the Hyperscale rpm spec for both CentOS 8 Stream and CentOS 9 Stream. When a build fails, Gitlab sends an email, notifying us of the problem so we can swiftly adapt the rpm spec to account for any new changes in the upstream build system. The daily builds are pushed to the testing repositories, allowing users to easily pull them in by enabling the Hyperscale testing repositories.
Kernel
Userspace utility packages such as kernel-tools (cpupower etc.) and perf have traditionally been built as part of the kernel source package on CentOS Stream, RHEL, and derivatives, and similarly we did this for the Hyperscale kernel.
This means that making any packaging change to these tools require rebuilding the entire kernel, and that the version number on the package does not really reflect the actual version given there are rebases from newer kernel versions.
We now have a separate kernel-tools source package, based on Fedora’s, that let us build these utilities separately. The ‘hs’ variant has minimal changes to get it to build on CentOS Stream 8 and CentOS Stream 9.
Container images
Our container build pipeline is fully automated, and container images are built on the CentOS OpenShift CI/CD infrastructure and published weekly on Quay.
We provide both CentOS Stream 8 and CentOS Stream 9 variants at quay.io/centoshyperscale/centos.
Spin images
We’ve made progress on the development of the Hyperscale spin. We’ve worked upstream with the kiwi project to support our needs. We were able to make our first prototype OpenStack cloud image in CBS with this, and are working through getting live media working properly. We hope to have initial release cloud and workstation images soon.
Package updates
An updated build of rasdaemon with support of unified SEL logging is now available for CentOS Stream 8 and CentOS Stream 9. An updated build of mstflint for CentOS Stream 9 adds the mstreg and mstlink tools. In the Experimental repo, createrepo_c has been rebased to include the latest upstream changes from CentOS Stream 9.
The automated pipeline for package updates notifications is now fixed and is once again producing notification tickets.
Intel repository
We maintain a hyperscale-intel repository with optimized versions of packages. As part of this work, we have released an optimized version of the zlib package for both CentOS Stream 8 and CentOS Stream 9 yielding significant performance gains.
DNF/RPM stack with CoW support
The Copy-on-Write stack was rebuilt on top of the latest packaging changes in upstream CentOS.
We’re still in the process of reworking the CoW patchset to address the latest upstream feedback; once the discussion has settled we will publish an updated version.
Health and Activity
The SIG continues to maintain a healthy development pace.
Meetings
The SIG holds regular bi-weekly meetings on Wednesdays at 16:00 UTC. Meetings are logged and the minutes for past meetings are available.
The SIG uses the #centos-hyperscale:fedoraproject.org Matrix room for for ad-hoc communication and work coordination; this room used to be bridged to the #centos-hyperscale IRC channel, but the bridge is currently inoperative. For async discussions and announcements we generally use the centos-devel mailing list. The SIG also holds open monthly video conference sessions to promote collaboration and social interaction.
Conference talks
During the last quarter members of the CentOS Hyperscale SIG attended several conferences where they were able to share the work the SIG is doing and meet up in person.
- Anita, Davide, Michel, Manu and Quentin attended LSFMMBPF 2023 and Open Source Summit North America in Vancouver, BC
- Anita, Daan, Davide, Neal and Richard attended DevConf.CZ 2023 in Brno, Czechia; Davide presented a talk covering Meta’s usage of Fedora ELN which touched on Hyperscale as well; we also had several conversations with the RPM and DNF developers about the next steps for RPM CoW
- Davide and Neal attended Flock 2023 in Cork, Ireland, and presented an update on SIG activities in the co-located CentOS Connect track. Hyperscale was also mentioned in the Alternative Images SIG update and during the Upstream collaboration in Enterprise Linux panel.
We have a page tracking conference presentations around Hyperscale-related topics. You can find references there to all talks mentioned below, including video recordings where available.
Conferences aren’t just about presentations though. The “hallway track” provides a great opportunity for serendipidous connection, and the various social events are often a great venue for folks to mingle and get to know each other in an informal setting. If you’d like to meet us in person at a future event please reach out.
In the remainder of the year we’re planning to attend All Systems Go 2023 and LFNW2023. Next year we expect to have a presence at FOSDEM 2024 and SCALE 21x, plus other events down the road.
Live streams
The SIG periodically does work live on Twitch from its official Twitch channel. Interested parties who want to watch and interact with us as we do work should follow us on Twitch to get notified for when we stream.
Planned work
The SIG tracks pending work as issues on our Pagure repository. Notable projects currently in flight include:
- using CBS to build our spin images
- shipping an updated QEMU package in EPEL
- integrate btrfs transactional updates as an optional feature
Issues for the Board
We have no issues to bring to the board’s attention at this time.