At Government Digital Service (GDS) we use Concourse for continuous delivery and other automation tasks across many of our projects. Concourse has a lot of community-contributed content, mostly pre-built container images published as public images on Docker Hub.
To help us in the Cyber Security team provide simple health monitoring for our pipelines, we built a webhook in AWS Lambda. We used one of these community-built container images, in this case http-api-resource, to send data to the webhook.
However we soon found that the http-api-resource image was on a published list of vulnerable containers. This list is an analysis of the top 1,000 most popular images on Docker Hub by number of downloads.
The container had a number of vulnerabilities marked as critical. The Cyber Security team investigated the vulnerabilities and discovered there was nothing wrong with the container code itself. All the vulnerabilities were all in the Python 3.6 base image.
We reviewed the vulnerabilities with GDS penetration testers and concluded that although critical, our exposure was limited given the role the container played in our pipeline.
To patch the vulnerabilities we used a multi-stage Docker build. This meant we could take the code from the latest version of the source image and then replace the base image with a less vulnerable Alpine base image.
We tested the new version with trivy, which is the same tool used to produce the vulnerable containers report. Happily, the report gave us the all clear, and we had a clean container to work with.
This means we now have our own version of the container running the same code, but with none of the security vulnerabilities.
Making the container public
The container is still pinned to a Python version so it’s not a perfect solution, since it will drift as the new base image gets older. This means as future versions of Python are released the container will stay on the older version. One approach might be to take the `latest` tag of Alpine and install Python ourselves rather than using a Python base image, although this has its own maintenance requirements.
Once we were happy with the new, clean container, we pushed the new image back to Docker Hub and then swapped our pipelines over.
A while later we noticed that the container had more than 100,000 downloads. We realised that what we’d done to fix the vulnerable container was probably useful for other people to know as well. The original container has over 200 million downloads on Docker Hub. The image is already public so we thought we’d write about it.
We’re now looking to take a similar approach to scanning other containers we build and use to see if we can patch vulnerable base images in a similar way.
Because we’ve not changed anything in the original source image you can implement it in exactly the same way. The only thing you need to change is the image repository path.
- name: http-api
If you want to implement a similar technique yourself here’s the Dockerfile as an example. The only thing we really needed to do is reinstall the Python requirements file to reinstall the runtime dependencies.
COPY --from=0 . .
RUN pip install --no-cache-dir -r requirements.txt
COPY --from=0 /opt/resource/ /opt/resource/
COPY --from=0 /opt/resource-tests/ /opt/resource-tests/
You can check out the image on Docker Hub. If you’ve used the image, please let us know your thoughts by leaving a comment below.