What is a Python Virtual Environment?
A Python virtual environment is an isolated Python environment that allows you to maintain separate dependencies for different Python projects. Setting up a Python virtual environments for each project ensures that it will have the requirements it needs without interference from other projects.
Why use a Python Virtual Environment?
When using Python, you might need to install packages and modules that aren't a part of the standard library. While doing a 'pip install –user some_package' will allow you to install Python packages in your home directory, you may run into issues later on with package dependencies. For example, if you are working on two different projects and Project 1 requires version 1.0 of a library whereas Project 2 requires version 2.0 of that same library, installing the dependency for Project 2 will likely break Project 1. An easy way around this is to create a separate virtual environment for each of your projects. The virtual environments are isolated from each other, allowing you to install dependencies for one project without worrying about breaking something in the other project.
Warning: Do not use "conda init" to initialize your shell for conda.
Some instructions or tutorials will tell you to run "conda init" to initialize your shell for conda. If you use NoMachine, DO NOT run "conda init", as that command will place conda initialization commands in your .bashrc that will prevent you from being able to establish a NoMachine session. For more information, please see our Problems Connecting With NoMachine page.
Creating a Python Virtual Environment
To create a virtual environment for your project, load the python module and then use the 'conda' command to create your virtual environment. In this example, we will create a Python 3.6 virtual environment named 'Project1'.
module load python/conda/3.6
conda create -n Project1 python=3.6
The 'conda create' command will output a list of packages that will be downloaded/installed and prompt you to proceed. Type 'y' to continue. This will install the Python version that you specified, along with the listed packages, to the .conda/envs/${env_name} directory in your home directory . In the above example, since the environment name we specified with the '-n' option to conda is 'Project1', the virtual environment will be installed into the .conda/envs/Project1 directory.
Please note that the version of python that you install into your virtual environment doesn’t need to be the same as the one you used in the ‘module load’ command. We could just as easily have loaded the python 3.6 module and then, for example, created a python 3.7 virtual environment as follows:
module load python/conda/3.6
conda create -n Project2 python=3.7
You may notice that when you create your virtual environment, the list of packages that conda installs is rather minimal. The virtual environments created in the above examples are 'barebones' python installs so you may need to install additional packages (e.g., numpy, pandas, etc.), especially if you are used to using Anaconda Python and the multitude of packages that it comes with. Information on how to install packages to your virtual environment can be found further down in this page in the section for installing packages.
Using the Virtual Environment
To use the virtual environment, load the python module (if you don't already have it loaded) and activate the environment.
module load python/conda/3.6
source activate Project1
After activating the virtual environment, the command prompt will change to indicate the conda environment you are in by prepending the name of your project, enclosed in parentheses. In this example, (Project1) will be prepended to your prompt. Once you have activated the environment, you can install packages and run any Python scripts associated with the project.
Deactivating the Virtual Environment
When you are done working in your virtual environment, you will want to deactivate it using the 'deactivate' command. Your prompt will return to normal, indicating your virtual environment is no longer active and your PATH and shell variables have been returned to normal.
source deactivate
Putting it all Together
conda create -n Project1 python=3.6
source activate Project1
source deactivate
conda create -n Project2 python=3.7
source activate Project2
source deactivate
Listing your Virtual Environments
To see a list of the Python virtual environments that you have created, you can use the 'conda env list' command. This command will give you the names as well as the filesystem paths for the location of your virtual environments.
conda env list
Note that in the list of environments, one of the environments will have an asterisk (*) by it. This is used to denote the environment that is currently active.
Listing Packages Installed in a Virtual Environment
After activating a virtual environment, if you want to see what packages are installed, you can use the 'conda list' command list the packages as well as their versions.
source activate Project1
conda list
Installing Packages to a Virtual Environment
Virtual environments are created with a barebones site library when not linked to the base Anaconda installation. Unless your code only relies upon the Python standard library, you will want to use either the conda or pip package managers to install additional libraries.
Installing a package using pip
To use pip to install a package to your virtual environment, just activate the environment (if you haven't already done so) and use the 'pip' command to install the package. In this example, we will install the numpy package to the Project1 virtual environment.
source activate Project1
pip install numpy
Installing a package using conda
There are a few different ways to use conda to install a package into your virtual environment, the easiest of which is to activate the environment and then use the conda command to install the package.
source activate Project1
conda install numpy
You can also install a package without activating the environment by specifying the environment name in the conda command.
conda install --name Project1 numpy
Differences between conda and pip
The differences between conda and pip are often misunderstood as both package managers provide similar functionality. On one hand, pip is well-adopted in the Python community and the Python Package Index (PyPI) contains a vast array of packages maintained by the community. On the other hand, Conda is a general-purpose package manager, can help track dependencies for non-Python libraries, and is fully compatible with pip. In general, each package manager compliments the other and you are free to use whichever provides the package you wish to install.
Common Commands
| Function | Conda | Pip |
| Search for a package in the default repo | conda search {package} | pip search {package} |
| Install a package | conda install {package} | pip install {package} |
| List installed packages | conda list | pip list |
| Upgrade a package | conda upgrade {package} | pip install --upgrade {package} |
| Remove a package | conda remove {package} | pip uninstall {package} |
Conda notes
binstar to search public channels
Pip notes
Best practice: requirements file
Value: reproducibility; pinning package versions.
With pip:
pip freeze > requirements.txt
With conda (similar to pip):
conda list -e > requirements.txt
Conda + pip:
conda env export > requirements.yaml
Deleting a Virtual Environment
When you are done with a project and have no further need for the associated Python virtual environment, you can delete it with the 'conda remove' command.
conda env remove -n Project1
Note: if you get an error that states “cannot remove current environment. deactivate and run conda remove again”, make sure that you have deactivated the environment. If the environment has been deactivated and you still get the same error, use the 'p' switch to the 'conda remove' command to specify the path to the virtual environment. (You can get the paths for all of your virtual environments with the 'conda env list' command.)
conda env remove -p /path/to/virtual/env