How to call pip programatically from Python

I needed this for the datasette install and datasette uninstall commands, see issue #925.

My initial attempt at this resulted in weird testing errors (#928) - while investigating them I stumbled across this comment in the pip source code:

# Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.
# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.
# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.
# The following example shows how to use runpy to invoke pip in that
# case:
#
#     sys.argv = ["pip", your, args, here]
#     runpy.run_module("pip", run_name="__main__")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.

So I did that! Here's the working version of my datasette install command:

 @cli.command() 
 @click.argument("packages", nargs=-1, required=True) 
 def install(packages): 
     "Install Python packages - e.g. Datasette plugins - into the same environment as Datasette" 
     sys.argv = ["pip", "install"] + list(packages) 
     run_module("pip", run_name="__main__") 

And here's how I wrote a unit test for it:

@mock.patch("datasette.cli.run_module")
def test_install(run_module):
    runner = CliRunner()
    runner.invoke(cli, ["install", "datasette-mock-plugin", "datasette-mock-plugin2"])
    run_module.assert_called_once_with("pip", run_name="__main__")
    assert sys.argv == [
        "pip",
        "install",
        "datasette-mock-plugin",
        "datasette-mock-plugin2",
    ]

Related

• homebrew Packaging a Python CLI tool for Homebrew - 2020-08-11
• python Packaging a Python app as a standalone binary with PyInstaller - 2021-01-04
• python Installing and upgrading Datasette plugins with pipx - 2020-05-04
• datasette Writing a CLI utility that is also a Datasette plugin - 2022-11-21
• pytest Using pytest and Playwright to test a JavaScript web application - 2022-07-24
• pytest Writing pytest tests against tools written with argparse - 2022-01-08
• datasette Writing Playwright tests for a Datasette Plugin - 2024-01-08
• pytest Registering temporary pluggy plugins inside tests - 2020-07-21
• readthedocs Running pip install '.[docs]' on ReadTheDocs - 2023-11-24
• electron Bundling Python inside an Electron app - 2021-09-08

Created 2020-08-11T17:07:25-07:00 · Edit