Over past few months, we've been working on CentOS Community Container Pipeline which aims to help developers focus on what they love doing most - write awesome code - and sysadmins have an insight into the image by providing metadata about it! The project code is hosted at Github.com since its inception. The hosted service, that runs off this code, is available to the community at large, and delivers content to registry.centos.org.
What is CentOS Community Container Pipeline?

CentOS Community Container Pipeline enables developers and sysadmins to have a container images built, tested and scanned on the CentOS Project's infrastructure right after a developer pushes code to the git repository!

Container Pipeline Flow

Once the developer pushes code to git repo, Container Pipeline fetches the changes and container images are built using OpenShift which provides an enterprise distribution of Kubernetes project. Once the image is built, it gets scanned using atomic scanners (more on this soon!). The result of these scanners is combined into a mail and sent to the author of the container image. Container images can also be tested using the user provided test scripts to ensure that container can be spinned off the image on platforms like CentOS Linux, CentOS Atomic Host and OpenShift.

Why scan images?

Building container images and spinning containers is rather simple. Having more information a.k.a metadata about the container images before running them in one's production environment is of paramount value! Of course, the kind of information is what makes it of paramount or negligible value. That's what we aim to provide with CentOS Community Container Pipeline.

Scanners in CentOS Community Container Pipeline

At this point we have two scanners operational. One that checks your CentOS Linux based container images for package updates and other that verifies them. Both the scanners are based on atomic tool developed by the Project Atomic folks. We are working on rolling out more scanners in near future!

Atomic Scanner

The scanners based on atomic are run automatically by the Pipeline after successful completion of image building process. These scanners can be run stand-alone as well! That is, you can install the scanner on your CentOS Linux based systems and run it against a container image built on CentOS Linux base image. And it does this without bringing up or executing the container itself.

In the pipeline, upon completion of scan process, the user is notified about issues with the image that need to be addressed. Addressing these issues would instill more confidence in deploying the resulting container image in a production environment.

Besides scanning an image after it is built, in near future, scanners would also run periodically and provide developer with the actionable information.

yum update scanner

This scanner provides user with the information about RPM packages that need to be updated in the container image. If you're a developer this information is helpful to ensure you're running latest packages with bug and security fixes to avoid having surprises in production.

Example output:

$ atomic scan --scanner pipeline-scanner --rootfs /mnt registry.centos.org/centos/centos
...

Files associated with this scan are in /var/lib/atomic/pipeline-scanner/2016-11-10-10-30-46-609885.

Scanner ran succesfully and has stored the scan data under /var directory. Let's see the output:

$ cat /var/lib/atomic/pipeline-scanner/2016-11-10-10-30-46-609885/_mnt/image_scan_results.json
{
    "Scanner": "pipeline-scanner", 
    "Successful": "true", 
    "Start Time": "2016-11-10-10-42-46-265018", 
    "Scan Results": {
        "Package Updates": [
            "bind-license.noarch", 
            "kmod.x86_64", 
            "kmod-libs.x86_64", 
            "kpartx.x86_64", 
            "openssl-libs.x86_64", 
            "python.x86_64", 
            "python-libs.x86_64", 
            "systemd.x86_64", 
            "systemd-libs.x86_64", 
            "tzdata.noarch"
        ], 
        "OS Release": "CentOS Linux 7 (Core)"
    }, 
    "Scan Type": "Image Scan", 
    "CVE Feed Last Updated": "NA", 
    "Finished Time": "2016-11-10-10-42-52-184442", 
    "UUID": "mnt"
}

The Package Updates key in above output lists packages that need to be updated in the scanned container image.

RPM verify scanner

As its name suggests RPM verify scanner verifies all installed files (libraries and binaries) via RPM packages in given container image. It reports any modified or tampered libraries and binaries in given container image. This is useful to ensure that given container image is not shipped with any tainted libraries or binaries.

Example output:

$ atomic scan --scanner rpm-verify docker.io/centos/postgresql
{
    "Scanner": "scanner-rpm-verify",
    "Successful": "true",
    "Start Time": "2016-11-10-19-49-06-740445",
    "Scan Results": {
        "rpmVa_issues": [
            {
                "config": false,
                "issue": "missing",
                "rpm": {Once the developer pushes code to git repo, Container Pipeline fetches the changes and container images are built using OpenShift which provides an enterprise version of Kubernetes project. Once the image is built, it gets scanned using atomic scanners (more on this soon!). Container images can also be tested using the user provided test scripts to ensure that container can be spinned off the image on platforms like CentOS Linux, CentOS Atomic Host and OpenShift.
                    "VENDOR": "CentOS",
                    "PACKAGER": "CentOS BuildSystem ",
                    "BUILDHOST": "worker1.bsys.centos.org",
                    "RPM": "glibc-2.17-55.el7_0.1.x86_64",
                    "SIGNATURE": "RSA/SHA256, Sat Aug 30 02:20:20 2014, Key ID 24c6a8a7f4a80eb5"
                },
                "filename": "/sbin/sln"
            },
            {
                "config": false,
                "issue": "........P",
                "rpm": {
                    "VENDOR": "CentOS",
                    "PACKAGER": "CentOS BuildSystem ",
                    "BUILDHOST": "worker1.bsys.centos.org",
                    "RPM": "iputils-20121221-6.el7.x86_64",
                    "SIGNATURE": "RSA/SHA256, Fri Jul  4 07:38:44 2014, Key ID 24c6a8a7f4a80eb5"
                },
                "filename": "/usr/sbin/clockdiff"
            }
        ]
    },
    "Scan Type": "RPM Verify scan for finding tampered files.",
    "CVE Feed Last Updated": "NA",
    "Finished Time": "2016-11-10-19-49-10-933952",
    "UUID": "da4ffaac638fada8723c6721721d99b0dfaba67d79c8507e881ee8327e17ecb"
}

Adding your container to the pipeline

It's simple! Add an entry for your opensource project under index.d directory on CentOS Container Index. You can see a few files representing projects or individual developers under this directory already. Also, you need to have a cccp.yml file in your project that has information useful for the Container Pipeline to use. You can refer respective GitHub repos to get more information. Or get in touch with us on #centos-devel IRC channel on FreeNode network.

Dharmit Shah and Navid Shaikh