Managing Python Packages with Conda Environments on the Cluster

Conda environments are an excellent method for managing python packages and libraries in the cluster environment. While a version of Anaconda3 is available as a software module, this package can cause path issues and prevent other modules from working correctly. The preferred method of managing conda environments is to use Miniconda. Miniconda is a lightweight program which can be installed directly to your /home directory.

The following instructions detail the process of downloading and installing Miniconda and walk through how to create and activate conda virtual environments.

Installing Miniconda

Download miniconda3:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh

Install miniconda to your preferred path:

bash miniconda.sh -b -p $HOME/miniconda

Creating a Conda Virtual Environment

Once miniconda has been installed, you can create a custom conda environment. This can be done by manually creating an environment and adding programs to it, or by using a .yml file which includes a predefinied software environment.

To create a miniconda environment, first activate base conda environment:

source $HOME/miniconda/bin/activate

To create a conda environment named 'myenv':

conda env create -n myenv

Alternatively, if a specific version of python is required, you can specify the desired version by appending a 'python=version number' parameter. e.g.:

conda env create -n myenv python=3.12

Once you have created a virtual environment, can activate it and install necessary programs and libraries into the environment:

conda activate myenv

conda install <program1>
conda install <program2>

Creating a Conda Virtual Environment using a yml file

Many github repositories will include a .yml (or similar) file which provides a list of the required software and libraries for a given program. To create a conda environment which includes all of these files, download the yml file, and then run 'conda env create -f filename'.

For example, the following commands will create an environment for MAKER 3, a set of tools for genome annotation:

wget https://wiki.orc.gmu.edu/mkdocs/maker/maker_3.01.03_environment.yml
conda env create -f maker_3.01.03_environment.yml

When creating a conda environment from a file, conda will list the name of the newly created environment. You can then activate that environment. If you forget the name of your conda environments, you can list them with the command:

conda env list

Using Programs installed in a Conda Virtual Environment

In order to use the programs installed in a virtual environment the environment must be activate. To do this, first activate the base conda environment and then activate your desired virtual environment. This must be done whenever logging in to a new node, this process is also necessary in any slurm scripts which run programs using the environment.

Activate base environment.

source $HOME/miniconda/bin/activate

Your terminal prompt should now indicate that the base conda environment is active, e.g: (base) [username@computername ~]$

Activate an environment named 'myenv'

conda activate myenv

Your terminal prompt should then display the name of your active environment: (myenv) [username@computername ~]$

You can now use any programs installed to that virtual environment.