Azure Pipelines and Python Virtual Environments

Virtual environments in Python can cause quite a few problems with Windows-based Azure Pipelines. From unreported errors, to ModuleNotFoundErrors, to screwed up environment variables, the issues are thorny but here are a few tips to help smooth them out.

The problem is the activate script

Normally, to create a virtual environment on Windows in Python, you do these three steps:

1. Install virtualenv
2. Call virtualenv, and
3. Call the activate script.

Your commands might look something like this:

C:\> python -m pip install --upgrade setuptools virtualenv wheel
C:\> virtualenv C:\venv
C:\> C:\venv\Scripts\activate.bat
(venv) C:\> 

The change in your prompt is a reminder that you are now using the virtual environment.

Unfortunately, this normal way of doing things causes problems with Azure Pipelines. In specific, the activate script kills the step where it is called. After calling activate, you won’t see any output from your step. Even worse, Azure Pipelines gives no indication that anything is wrong. In my testing, it seems that Pipelines doesn’t even execute commands after the activate call.

For example, consider this snippet from an azure-pipelines.yml:

  - script: |
      mkdir $(venv)
      $(buildPython.pythonLocation)\python -m virtualenv $(venv)
    displayName: setup virtual environment
  - script: |
      $(venv)\Scripts\activate.bat
      echo "This is really bad."
      echo "A big error has occurred!!"
      exit 45
    displayName: setup virtual environment

In the second step, the echoed messages are never displayed in the Azure Pipelines console and the step exits error free. Because of that and because with a CI pipeline we’re often looking for errors rather than to make sure everything ran, it is easy to miss.

Until Azure Pipelines and virtualenv work better together, we’ll have to use the virtual environment without using the activate script.

Fix #1: Use call!

If you’re going to execute a batch script in an Azure Pipelines script step, you must use the call command. This basic point seems to be significantly underemphasized in the main documentation but it is covered.

Without using call, you get no output from the batch script, it doesn’t run, there are no errors, and everything after the script doesn’t run. Since npm is a batch script on Windows, I’m guessing a few JavaScript developers have run into this issue as well.

The snippet above can be fixed like this:

  - script: |
      mkdir $(venv)
      $(buildPython.pythonLocation)\python -m virtualenv $(venv)
    displayName: setup virtual environment
  - script: |
      call $(venv)\Scripts\activate.bat
      echo "This is really bad."
      echo "A big error has occurred!!"
      exit 45
    displayName: setup virtual environment

Fix #2: Set up your own environment

Alternatively, if you just want to skip the activate batch script, you can set up everything on your own. From the perspective of a virtual environment user, activating a virtual environment just means changing the environment a little to point to the virtual environment rather than the regular environment. You only need to set three environment variables to “activate” the virtual environment:

1. PATH must have the virtual environment directory first
2. VIRTUAL_ENV should exist and contain the virtual environment directory
3. PYTHON_HOME needs to be blanked out

Manually activating a virtual environment may not be 100% future proof if the virtualenv maintainers decide to use more or different environment variables later, but this is the best solution I’ve found. (Comment below if you’ve found a better way.)

This snippet of an azure-pipelines.yml shows an easy way to create a step that uses a virtual environment:

  variables:
    venv: $(System.DefaultWorkingDirectory)\venv
  steps:
  - script: $(buildPython.pythonLocation)\python -m pip install --upgrade setuptools virtualenv wheel
    displayName: install python basics
  - script: |
      mkdir $(venv)
      $(buildPython.pythonLocation)\python -m virtualenv $(venv)
    displayName: set up the virtual environment
  - script: |
      set Path=$(venv)\Scripts;%Path%
      echo "We're in a virtual environment!"
      pip install --editable .
    displayName: do things in the virtual environment
    env:
      PYTHON_HOME: ""
      VIRTUAL_ENV: $(venv)

Remember that environment variables do not persist from step to step. However, job- or global-level variables are available across steps.

In the example above, the job-level variable venv is available in all the steps butPATH, PYTHON_HOME, and VIRTUAL_ENV would not be set in a theoretical fourth step after these three.

Multiple steps can use the same virtual environment, but you’ll need to set the step-level environment variables in each.

Follow us on Twitter 🐦 and Facebook 👥 and join our Facebook Group 💬.

To join our community Slack 🗣️ and read our weekly Faun topics 🗞️, click here⬇

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇