Use packages from private repositories with pip

As it is often required and many people seem to have this issue, the following describes an easy solution to include packages from private repositories in the requirements.txt.

For this example, a package depends on another package, that is under version control in a private Gitlab repo. A deploy token will be used, although one could also authenticate using a key pair and use git+ssh. Another alternative would be to make use of Gitlabs PyPI registry functionality. A detailed description is available here.
At first, you need to create a deploy token in Gitlab. You can find the options under Settings/Repository.

Specify the name of the token and select the read_repository permissions. Afterward, click on Create deploy token and note the username and respective token.

You can now add the package to your requirements.txt like this:

git+https://gitlab+deploy-token-XXXX:YOURTOKEN@gitlab.com/YOURGROUP/YOURPACKAGE.git@master#egg=PACKAGE

The setup.py of the package that depends on your private packages should then look like the following. It processes the requirements.txt line by line, separating PyPI packages and private dependencies. Later are added as dependency links to setuptools, which take care of installing them.

!/usr/bin/env python 
import os 
try: 
    from setuptools import setup 
except ImportError: 
    from distutils.core import setup 
# Preprocess requirements.txt
with open('requirements.txt', 'r') as r: 
    private = [] 
    public = [] 
    requirements_lines = r.read().splitlines() 
    for line in requirements_lines: 
        if line.startswith('#'): 
            # Skip comments 
            continue 
        elif 'git' in line: 
            private.append(line) 
        else: 
            public.append(line) 
 
with open('README.rst', 'r') as r: 
    readme = r.read() 
 
setup( 
    name='package', 
    packages=["package"],
    version='0.1', 
    author="Linus Kohl", 
    author_email="linus@munichresearch.com", 
    python_requires='>=3.6', 
    license='GPLv3', 
    long_description=readme, 
    # Important
    install_requires=public, 
    dependency_links=private 
)
    install_requires=public, 
    dependency_links=private 
)