Python Virtualenvwrapper

Neil Shephard

13 August 2024 13:00

This article introduces virtualenvwrapper for creating and working with Python Virtual Environments showing how to install and use it and highlighting some useful features.

Photo by Neil Shephard.

If you use Python regularly chances are you are familiar with and use virtual environments to isolate the installed packages as you develop your code from the Python packages installed at the system level. The standard library venv, and other packages such as pipx, virtualenv and Anaconda are all popular options. Many of the popular package development suites such as Hatch, PDM and Poetry will also manage virtual environments for you too. One you might not have come across is virtualenvwrapper which is…

a set of extensions to Ian Bicking’s virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

NB If you are using Microsoft Windows you can follow along by using Windows Subsystem for Linux, otherwise virtualenvwrapper installation and usage is slightly different under Windows and you should consult the virtualenvwrapper.

What are Virtual Environments?

Before getting started a quick recap for readers who aren’t familiar with virtual environments.

When you run programmes, such as invoking python at the command line, whether that is GNU/Linux, Mac OSX or Microsoft Windows, the operating system searches for the command you run in a defined set of places. It doesn’t look everywhere, rather it looks in the locations defined by an environment variable called $PATH. This variable holds a set of paths to locations in which to look for executable/binary programmes. You can inspect its value with echo $PATH. The order in which the listed paths are checked takes left-to-right precedence and on my system this looks like the following.

❱ echo $PATH
/home/neil/.local/share/pnpm:/home/neil/bin:/home/neil/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/18/bin:/usr/lib/llvm/17/bin:/home/neil/.local/bin:/home/neil/.cargo/bin:/home/neil/.node/bin

Thus if I invoke python the first path that is checked for a binary with that name is /home/neil/.local/share/pnpm. If one isn’t found the second is checked (/home/neil/bin), then the third (/home/neil/.cargo/bin) and so on until a match is found. If no match is found you can check the path to a given binary using which and if no programme with that name is found you are told. This means its either not installed o your system or you perhaps made a tpyo (Hint use tab-complete its usually installed and works in most shells).

❱ which python
/usr/bin/python
❱ which ptyhon
ptyhon not found

A virtual environment modifies the $PATH variable, placing a new path at the start that points to a directory under your user account that contains a copy of python and the specific set of packages you have installed under that environment. We’ll cover this in more detail but if I look at the $PATH variable after having activated the topostats virtual environment you will see that its value has been modified and the first directory that is checked is /home/neil/.virtualenvs/topostats/bin and if I check which version of python is used it is the one within this directory.

❱ echo $PATH
/home/neil/.virtualenvs/topostats/bin:/home/neil/.local/share/pnpm:/home/neil/miniconda3/condabin:/home/neil/bin:/home/neil/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/18/bin:/usr/lib/llvm/17/bin:/home/neil/.local/bin:/home/neil/.cargo/bin:/home/neil/.node/bin
❱ which python
/home/neil/.virtualenvs/topostats/bin/python

Managing Virtual Environments with virtualenvwrapper

As the name suggests virtualenvwrapper is a wrapper for the virtualenv package that adds a bunch of convenience functions and features that, in this author’s view, make it easier to install, activate/deactivate and manage your virtual environments.

Installation

I recommend installing virtualenvwrapper at the system level, ideally with your systems package manager. Under different GNU/Linux or OSX using Homebrew this can be done with…

❱ sudo pacman -Syu virtualenvwrapper       # Arch
❱ sudo emerge -av virtualenvwrapper        # Gentoo
❱ sudo apt-get install virtualenvwrapper   # Debian/Ubuntu
❱ brew install virtualenvwrapper           # OSX Homebrew

…or if you are happy installing packages using pip you can follow the official installation instructions and use pip to install at the system level.

❱ sudo pip install virtualenvwrapper

Alternatively you can install locally using pip by using the --user flag.

❱ pip install --user virtualenvwrapper

You now need to ensure that the virtualenvwrapper.sh is sourced each time you start a new shell. To do this you need to add lines similar to the following to your ~/.bashrc or ~/.zshrc files depending on which shell you use (the former is the default on most GNU/Linux systems and Windows Subsystem for Linux, the later is the default under OSX).

# Configure and setup virtualenvwrapper
export WORKON_HOME=${HOME}/.virtualenvs
export PROJECT_HOME=${HOME}/work/
source /usr/local/bin/virtualenvwrapper.sh

The values of PROJECT_HOME should be specific to where you wish to create your projects, whilst the argument to source will depend on how you have installed virtualenvwrapper, you can check using which so use the value returned by

❱ which virtualenvwrapper.sh
/usr/bin/virtualenvwrapper.sh

Always using a virtual environment

Once in virtualenvwrapper is installed you can change a configuration option to pip to ensure that you are always using a virtual environment when you attempt to install packages using pip by adding the following to the ~/.config/pip/pip.conf file on your system.

[global]
require-virtualenv = True

Creating a Virtual Environment with virtualenvwrapper

This is straight-forward.

❱ mkvirtualenv test_env
(test_env) ❱

By default on creating a new virtual environment it is activated and your prompt should change to reflect this with the environment name appearing as part of the prompt. You can check that the python that will be executed resides in this directory using which.

(test_env) ❱ which python
/home/neil/.virtualenvs/test_env/bin/python

Activating/Deactivating Virtual Environment

You have been dropped into the test_env virtual environment. To exit from it use the deactivate command, again you can check which version of python is found on your $PATH using which.

(test_env) ❱ deactivate
(test_env) ❱ which python
/usr/bin/python

You won’t always create a virtual environment when you want to use it though, that would be a waste of time and bandwidth downloading and installing packages (although these are cached and used if no new updates are available). To activate an existing virtual environment you use the workon command.

 ❱ workon test_env
(test_env) ❱

Project Directories

Typically code for a project resides in its own directory and this can be automatically bound to the virtual environment using the mkproject command instead of mkvirtualenv. The project directory is stored in the $PROJECT_HOME path you will have configured during installation. You can then create a project and a virtual environment with…

❱ mkproject new_project
(new_project) ❱

One advantage this confers is that when you activate the project, e.g. with workon new_project your working directory will automatically change to that of the project, i.e. $PROJECT_HOME/new_project, meaning you don’t have to use cd to change directory to that location.

Manually setting Project Directories

If you don’t use mkproject to create a project you can set it manually after activating a virtual environment using setvirtualenvproject in the desired directory. This adds an entry to the ~/.virtualenv/<env_name>/.project file that reflects the directory associated with the environment.

❱ mkvirtualenv new_project
(new_project) ❱ cd ~/work/some/other/path/for/new_project
(new_project) ❱ setvirtualenvproject

Disabling Switching to Project Directories

The environment variable VIRTUALENVWRAPPER_WORKON_CD “controls whether the working directory is changed during the post activate phase. The default is 1, to enable changing directories. Set the value to 0 to disable this behaviour for all invocations of workon”. If this is something you don’t want to happen you should set this in your ~/.bashrc or ./zshrc, otherwise the default can be left alone.

export VIRTUALENVWRAPPER_WORKON_CD=0

Removing Virtual Environments

You can remove a virtual environment with rmvirtualenv <env_name>.

One neat option if you want to keep a virtual environment but install all packages anew is the ability to remove all third-party packages in the current virtual environment using wipeenv. You can then use pip to reinstall a package’s dependencies or a set of packages from a requirements.txt file.

Temporary Virtual Environments

Sometimes you just want to try something out quickly in a clean virtual environment, if for example you are reviewing a Pull Request and want to check installation and tests pass. virtualenvwrapper can help here as it has the mktmpenv. There are two options here -c|--cd or -n|--no-cd which changes directory post-activation or doesn’t respectively. The environment gets a unique name and will be deleted automatically when it is deactivated.

virtualenvwrapper hooks

Just like the various hooks available in Git, virtualenvwrapper also supports hooks that allow scripts to be run in response to various events. These reside under your $VIRTUALENVWRAPPER_HOOK_DIR which by default is the same as your $WORKON_HOME directory and in a typical standard installation will be ~/.virtualenvs.

The available scripts that are recognised are…

• get_env_details
• initialize
• premkvirtualenv
• postmkvirtualenv
• precpvirtualenv
• postcpvirtualenv
• preactivate
• postactivate

Each of these is a simple shell script and will start with the scripting language to use e.g. #!/usr/bin/bash or #!/usr/bin/zsh depending on your shell. You can then script the actions you wish to take when the script is executed.

Install minimal requirements

I’m a big fan of dotfiles¹, mine are hosted on GitLab, it’s a repository of my configuration files and scripts that I use regularly across multiple computers. Because I’m lazy I wrote a couple of requirements.txt files for installing packages in my virtual environments.

• requirements.txt : holds everything I might ever want to use in Python.
• python-lsp-requirements.txt : Install packages for setting up a Python Language Server (which I use from Emacs).

• venv_minimal_requirements.txt

  a minimal set of the most common Python packages I am likely to want when creating a new virtual environment.

Because I have my dotfiles cloned to the same location on every computer (~/dotfiles) I added the following to the ~/.virtualenvs/postmkvirtualenv ² which will install all of the packages listed in ~/dotfiles/python/venv_minimal_requirements.txt whenever I create a new virtual environment, whether that is with mkvirtualenv or mktmpenv.

pip install --no-cache-dir -r ~/dotfiles/python/venv_minimal_requirements.txt

This ensures the latest versions of each package listed in ~/dotfiles/python/venv_minimal_requirements.txt were downloaded and installed as the --no-cache-dir prevents using cached versions of packages.

A smarter script

This served me well for a time, but occasionally I found I didn’t want to install any packages in a new virtual environment (most often when testing new branches using mktmpenv) and I’d have to remember to comment out the line in the hook file (~/.virtualenvs/postmkvirtualenv) before creating the environment. Typically though I’d forget to do this and would have to halt installation of required packages, deactivate the environment, then comment it out and create a new environment.

This quickly became irksome.

But ~/.virtualenvs/postmkvirtualenv is just a script and so we can use a bit of scripting knowledge to make it interactive and ask the user if they want to install the packages listed in venv_minimal_requirements.txt. I found a really useful answer on StackOverflow in the How do I prompt for yes/no/cancel input in a Linux shell script that showed several different ways to prompt the user for a response as to whether they want to do something.

I therefore updated my ~/.virtualenvs/postmkvirtualenv to the following which prompts for a numeric response, 1 for Yes and 2 for No and takes the appropriate action, installing using my original invocation of pip if I want to install packages and enter 1 or installing nothing if I enter 2.

#!/usr/bin/zsh
# This hook is sourced after a new virtualenv is activated.


echo "Do you wish to install minimal requirements (from venv_minimal_requirements.txt)? "
select yn in "Yes" "No"; do
  case $yn in
    Yes ) pip install --no-cache-dir -r ~/dotfiles/python/venv_minimal_requirements.txt; break;;
    No ) echo "No packages installed. install packages with 'pip'.\n"; break;;
  esac
done

NB You may want to tweak the opening shebang if you use the Bash shell.

Drawbacks

There are a couple of drawbacks I’ve found to using using virtualenvwrapper.

The first is that mkproject doesn’t allow nesting of project directories, you have to specify a single directory and it will be created under the $PROJECT_HOME directory with the associated environment name. This doesn’t work for me as I use the structure ~/work/git as the base but then have sub-directories based on the Git Forge (GitHub/GitLab/Codeberg) the repository is associated with and further nesting to reflect the user/organisation within as I have both a personal and work accounts. E.g. ~/work/git/hub/ns-rse/ns-rse.github.io which is the source (github.com/ns-rse/ns-rse.github.io) for my blog hosted in GitHub under my work account (ns-rse) or ~/work/git/lab/nshephard/tcx2gpx which is a project of mine (tcx2gpx) which is hosted on GitLab.

This means that if I wanted to create a project with mkproject based on $PROJECT_HOME being /work/git following this structure I would specify mkproject git/lab/new_project and whilst the directory is created, the virtual environment is created as git/lab/new_project which is truncated to git and you can’t workon git because the activation scripts are nested deeper under git/lab/new_project. Further each environment I created would then conflict. I could probably work around this by creating symbolic links but in practice I just use mkvirtualenv and setvirtualenvproject after I git clone work.

This is a problem specifically of my own creation though and I can live with it/work around it.

Something other users might find causes greater friction is that virtualenvwrapper doesn’t support creating and keeping the virtual environments within the project directory itself. This is never something that I’ve wanted to do myself though as I find it tidier to keep them all in one place and easier to find and remove obsolete environments.

Conclusion

There are many options for installing and managing Python Virtual Environments including those provided by package development frameworks such as Hatch, PDM and Poetry but virtualenvwrapper is a viable option that has many neat features, more than covered here, that make working with virtual environments straight-forward. The documentation is excellent and worth reading in detail if this introductory article has sparked your interest as there are more than options than those covered here.

NB This post is based on (and hopefully improves on) two earlier blogs made by the author. Most of the improvements are Thanks to my colleagues Tamora James and Pete Heywood who reviewed a draft and provided feedback.

• virtualenvwrapper
• virtualenvwrapper hooks

Links

• Anaconda
• Hatch
• PDM
• pipx
• Poetry
• venv
• virtualenv
• virtualenvwrapper