Using Python's pip Module Programmatically

by James Johnson

Everyone knows about Python’s pip utility. Did you know it’s also a standard python package that you can import and use at runtime? See examples using the pip module to install packages and parse requirements files.

Installing Packges with pip

pip is the standard python utility that everyone uses to install python packages. (if you’re still using easy_install, seriously why? Yes I’m actually serious and want to know, because I don’t get it):

pip install --user plumbum

You should totally check out the plumbum package! It’s like the sh package but cooler

You can also import the pip module directly in your python scripts and use it to install packages from within Python:

import pip

try:
    pip.main(["install", "--user", "plumbum"])
except SystemExit as e:
    pass

Why would you want to do this? I’ve needed to be able to install packages at runtime with the self-imposed requirement of not forking off yet-another-process just to install something. In my opinion being able to do it from within python is the cleaner approach.

Notice that I wrapped the pip.main(...) call in a try/except to catch a SystemExit exception. This is because the pip module explicitly exits when any errors occur during the installation process.

Parsing Requirements Files With pip

Requirements files can be much more complicated than most would initially think. Some simple and complicated requirement examples are below:

requests
FooProject >= 1.2 --global-option="--no-user-cfg" \
                  --install-option="--prefix='/usr/local'" \
                  --install-option="--no-compile"

-e git+https://github.com/d0c-s4vage/pyactlab.git@feather-master#egg=pyactlab
-e git+git@git.myproject.org:MyProject#egg=MyProject

FooProject == 1.2 --hash=sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 \
                  --hash=sha256:486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7a

Parsing requirements files yourself could be extremely complicated. Thankfully you can use the pip module itself to do it for you using the pip.req submodule:

import pip
import pip.req
import sys

def main(req_file):
    for item in pip.req.parse_requirements(req_file, session="somesession"):
        print("")
        if isinstance(item, pip.req.InstallRequirement):
            print("required package: {}".format(item.name))

            if len(str(item.req.specifier)) > 0:
                print("  " + str(item.req.specifier))

            if item.link is not None:
                print("  from: " + item.link.url)
                print("  filename: " + item.link.filename)
                print("  egg: " + item.link.egg_fragment)

            if len(item.options) > 0:
                for opt_type,opts in item.options.iteritems():
                    print("  {}:".format(opt_type))
                    if type(opts) is list:
                        for opt in opts:
                            print("    " + opt)
                    elif type(opts) is dict:
                        for k,v in opts.iteritems():
                            print("    {}: {}".format(k,v))

if __name__ == "__main__":
    if len(sys.argv) == 1:
        print("USAGE: {} <requirements_file>".format(sys.argv[0]))
        exit(1)

    main(sys.argv[1])

Running the requirements parsing example above on the requirements example yields the output:

required package: requests

required package: FooProject
  >=1.2
  install_options:
    --prefix='/usr/local'
    --no-compile
  global_options:
    --no-user-cfg

required package: pyactlab
  from: git+https://github.com/d0c-s4vage/pyactlab.git@feather-master#egg=pyactlab
  filename: pyactlab.git@feather-master
  egg: pyactlab

required package: MyProject
  from: git+git@git.myproject.org:MyProject#egg=MyProject
  filename: git+git@git.myproject.org:MyProject
  egg: MyProject

required package: FooProject
  ==1.2
  hashes:
    sha256: ['2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824', '486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7a']

If you are into these pip shenanigans, you’ll also probably be interested in hosting your own python package index. If so, check out the pip2pi module.

It may not be very often that you want to parse requirements.txt files, but the pip module provides a way for you to do it if you need to.

Most Interestingn Man and pip.req