Building Docker images that include private repositories

If you are building Docker images, it is often required to include code from private repositories. At risklio we needed to install Python packages from private Gitlab repositories via pip. The most important requirements were the following:

• No “special” Gitlab account
• Same credentials for each source repository
• Secure handling of keys, no leak into container

The solution came from Vladislav Supalov’s article about Multi-stage Builds and is adapted to Pythons package manager and Gitlab, which takes care of the keys.

Key generation

Start by creating a new RSA Key:

ssh-keygen -t rsa -b 4096 -C "deployment@yourcompany.com"

Then add the newly created public key to Settings / Repository / Deploy Keys for every private repository you want to access in the build process.

Now add the content of the private key into a newly created environmental variable named something like SSH_PRIVATE_KEY in Settings / CI/CD / Environment variables section of the repository you want to build.

Configuring the build job

If you are building your container using docker-compose, you can set the private key as an environmental variable and pass it like this:

# docker-compose.yml
version: '2'
services:
  microservice:
    build:
      context: .
      args:
        - SSH_PRIVATE_KEY=${SSH_PRIVATE_KEY}

To store the key inside a .env variable, copy the output of the following command and place it as after SSH_PRIVATE_KEY=

cat id_rsa_private_key| sed -e :a -e '/$/N; s/\n/\\n/; ta'

When using the GitLab CI environment, modify the .gitlab-ci.yml in your project root directory according to the following. Adding the force-rm parameter always removes intermediate containers.

build_image: 
 image: docker:git 
 services: 
 — docker:dind 
 variables: 
 DOCKER_DRIVER: overlay 
 script: 
 — docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com 
# passing the private key as a variable
- docker build --force-rm --build-arg SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" -t registry.gitlab.com/USER/PROJECT . 
 — docker push registry.gitlab.com/USER/PROJECT:latest 
 only: 
 — master

The Dockerfile

The following example installs the required tools and downloads the packages via pip in the intermediate image, then copies the packages to a fresh image and installs them.

# INSECURE intermediate image that will be deleted 
# 
FROM python:3.6-slim AS insecure 
MAINTAINER Linus Kohl <linus@riskl.io> 
 
# the private key with access to the repositories is available in 
# the enviroment 
ARG SSH_PRIVATE_KEY 
 
# install git and openssh-client 
RUN apt-get update && apt-get upgrade -y && \ 
    apt-get -y install --no-install-recommends \ 
    git \ 
    openssh-client && \ 
    rm -rf /var/cache/apk/* 
 
# setup private key and known_hosts file 
RUN mkdir -m0700 /root/.ssh/ && \ 
    echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa && \ 
    chmod 600 /root/.ssh/id_rsa && \ 
    ssh-keyscan gitlab.com >> /root/.ssh/known_hosts 
 
# create package directory 
RUN mkdir -p /pip-packages/ 
WORKDIR /pip-packages/ 
 
# download packages specified in requirements.txt 
COPY app/requirements.txt /tmp/requirements.txt 
RUN pip download --no-cache -r /tmp/requirements.txt 
 
# SECURE shippable image 
# 
FROM python:3.6-slim AS secure 
MAINTAINER Linus Kohl <linus@riskl.io> 
 
# set and create application path 
ENV INSTALL_PATH /feed-director 
RUN mkdir -p $INSTALL_PATH 
WORKDIR $INSTALL_PATH 
 
# copy packages from insecure, to secure image 
COPY --from=insecure /pip-packages/ /pip-packages/ 
# install the packages 
RUN pip install --no-index --find-links=/pip-packages/ /pip-packages/* 
# install the application 
COPY app . 
ENTRYPOINT ["python"] 
# run application 
CMD [ "run.py" ]

Conclusion

The private key will neither appear in the history of the final container, nor in the low-level information of it. Another benefit is the size of the container, as you could install and use build tools in the intermediate container without having to clean up everything.