Building Docker images that include private repositories
If you are building Docker images, it is often required to include code from private repositories. At risklio we needed to install Python packages from private Gitlab repositories via pip. The most important requirements were the following:
- No “special” Gitlab account
- Same credentials for each source repository
- Secure handling of keys, no leak into container
The solution came from Vladislav Supalov’s article about Multi-stage Builds and is adapted to Pythons package manager and Gitlab, which takes care of the keys.
Key generation
Start by creating a new RSA Key:
ssh-keygen -t rsa -b 4096 -C "deployment@yourcompany.com"
Then add the newly created public key to Settings / Repository / Deploy Keys for every private repository you want to access in the build process.
Now add the content of the private key into a newly created environmental variable named something like SSH_PRIVATE_KEY in Settings / CI/CD / Environment variables section of the repository you want to build.
Configuring the build job
If you are building your container using docker-compose, you can set the private key as an environmental variable and pass it like this:
# docker-compose.yml
version: '2'
services:
microservice:
build:
context: .
args:
- SSH_PRIVATE_KEY=${SSH_PRIVATE_KEY}
To store the key inside a .env variable, copy the output of the following command and place it as after SSH_PRIVATE_KEY=
cat id_rsa_private_key| sed -e :a -e '/$/N; s/\n/\\n/; ta'
When using the GitLab CI environment, modify the .gitlab-ci.yml in your project root directory according to the following. Adding the force-rm parameter always removes intermediate containers.
build_image:
image: docker:git
services:
— docker:dind
variables:
DOCKER_DRIVER: overlay
script:
— docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.com # passing the private key as a variable
- docker build --force-rm --build-arg SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" -t registry.gitlab.com/USER/PROJECT .
— docker push registry.gitlab.com/USER/PROJECT:latest
only:
— master
The Dockerfile
The following example installs the required tools and downloads the packages via pip in the intermediate image, then copies the packages to a fresh image and installs them.
# INSECURE intermediate image that will be deleted
#
FROM python:3.6-slim AS insecure
MAINTAINER Linus Kohl <linus@riskl.io>
# the private key with access to the repositories is available in
# the enviroment
ARG SSH_PRIVATE_KEY
# install git and openssh-client
RUN apt-get update && apt-get upgrade -y && \
apt-get -y install --no-install-recommends \
git \
openssh-client && \
rm -rf /var/cache/apk/*
# setup private key and known_hosts file
RUN mkdir -m0700 /root/.ssh/ && \
echo "${SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa && \
ssh-keyscan gitlab.com >> /root/.ssh/known_hosts
# create package directory
RUN mkdir -p /pip-packages/
WORKDIR /pip-packages/
# download packages specified in requirements.txt
COPY app/requirements.txt /tmp/requirements.txt
RUN pip download --no-cache -r /tmp/requirements.txt
# SECURE shippable image
#
FROM python:3.6-slim AS secure
MAINTAINER Linus Kohl <linus@riskl.io>
# set and create application path
ENV INSTALL_PATH /feed-director
RUN mkdir -p $INSTALL_PATH
WORKDIR $INSTALL_PATH
# copy packages from insecure, to secure image
COPY --from=insecure /pip-packages/ /pip-packages/
# install the packages
RUN pip install --no-index --find-links=/pip-packages/ /pip-packages/*
# install the application
COPY app . ENTRYPOINT ["python"]
# run application
CMD [ "run.py" ]
Conclusion
The private key will neither appear in the history of the final container, nor in the low-level information of it. Another benefit is the size of the container, as you could install and use build tools in the intermediate container without having to clean up everything.