Revamp CentOS Community Container Pipeline to run on OpenShift

Monday, 8, October 2018 Dharmit Shah announcement, builds, Community No Comments

It's been over a year since we published anything about the CentOS Community Container Pipeline. Many interesting things have happened during the past year, many things have changed and there's a complete shift in the architecture of the service that's was rolled out over the last weekend.

Wait, I've never heard of this project

If this is the first time you're hearing about CentOS Community Container Pipeline project, it would be best to refer this blog post, or the GitHub repo of the project, or the wiki page. But to put it in short, the service does below things:

How did the old system work?

When we talked about the project at DevConf.cz '18, we received a positive response from the audience. However, at that time, we knew that our service couldn't handle more build requests and on-boarding more community projects would be counter-productive when our backend didn't have the ability to serve those requests.

Old implementation of the service had a lot of plumbing. There are workers written for most of the features mentioned above.

Everything was spread across different hosts and we were using really huge Ansible playbooks to bring up the service. A fresh deployment took 30 minutes on an average. Testing any change in dev environment would require us to do a redeployment of the service which took another 15 minutes on an average. Deploying and maintaining this service was quite a pain!

What did we do about these problems?

Since long time we were discussing about developing our service on top of OpenShift. Then, at some point, we read about OpenShift Pipeline and found it interesting. We took the plunge and came up with a proof of concept implementation of CentOS Community Container Pipeline on top of OpenShift OKD using Minishift. Results were exciting! We were able to do parallel builds of container image, Jenkins Pipelines orchestrated the flow really well, build times were faster, we didn't need to use beanstalkd at all and, most importantly, there was very less code written to get things done!

With the POC in place, we went ahead with developing more tangible service on top of a real OpenShift cluster instead of developing on top of Minishift. What used to be individual workers doing their thing in old system is now pretty much all inside OpenShift Pipeline.

We now have an OpenShift Pipeline for every project on CentOS Container Index that does Pre-build, Dockerfile lint, container image build, scan the container image and push it to external registry; all from a single container! We have another OpenShift Pipeline for every project to do their weekly scans. So instead of having five workers to do these tasks and communicate with each other via beanstalkd, we have orchestrated things through OpenShift Pipelines.

What are we working on now?

We don't have Repo tracking implemented in the new architecture yet. We don't have a UI for the users to take a look at their build logs or weekly scan logs either. We're initially focusing on getting the UI for logs up and then we will start working on Repo tracking.  We are also working on setting up a CI job that tests core parts of the service on Minishift so that anyone willing to take the service for a spin should literally be able to do it on a Minishift VM!

Let us know your thoughts!

This project is solely focused on making things easier for open-source projects and its developers. If you are working on an open-source project that's building on top of CentOS, we would like to know your thoughts. If you need help getting started, you can contact us on IRC (#centos-devel on Freenode) or take a look at project documentation.

Dharmit Shah (dharmit on #centos-devel IRC)

Leave a Reply

Your email address will not be published. Required fields are marked *