Managing Python Packages with Conda Environments on the Cluster

Conda environments are an excellent method for managing python packages and libraries in the cluster environment. The preferred method of managing conda environments is to use Miniforge. Miniforge is a lightweight program which can be installed directly to your /home directory.

The following instructions detail the process of downloading and installing Miniforge and walk through how to create and activate conda virtual environments.

Installing Miniforge

Download Miniforge:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh

Install minifoge to your preferred path:

bash Miniforge3-Linux-x86_64.sh -b -p $HOME/miniforge

Creating a Conda Virtual Environment

Once miniforge has been installed, you can create a custom conda environment. This can be done by manually creating an environment and adding programs to it, or by using a .yml file which includes a predefinied software environment.

To create a miniforge environment, first activate base conda environment:

source $HOME/miniforge/bin/activate

To create a conda environment named 'myenv':

conda create -n myenv

Alternatively, if a specific version of python is required, you can specify the desired version by appending a 'python=version number' parameter. e.g.:

conda create -n myenv python=3.12

Once you have created a virtual environment, can activate it and install necessary programs and libraries into the environment:

conda activate myenv

conda install <program1>
conda install <program2>

To use programs installed to this environment on subsequent logins, acivate miniforge and the activate the environment:

source $HOME/miniforge/bin/activate
conda activate myenv

This can also be combined into a single command in order to directly activate both miniforge and a conda environment:

source $HOME/miniforge/bin/activate myenv

Creating a Conda Virtual Environment using a yml file

Many github repositories will include a .yml (or similar) file which provides a list of the required software and libraries for a given program. To create a conda environment which includes all of these files, download the yml file, and then run 'conda env create -f filename'.

For example, the following commands will create an environment for MAKER 3, a set of tools for genome annotation:

wget https://wiki.orc.gmu.edu/mkdocs/maker/maker_3.01.03_environment.yml
conda env create -f maker_3.01.03_environment.yml

When creating a conda environment from a file, conda will list the name of the newly created environment. You can then activate that environment. If you forget the name of your conda environments, you can list them with the command:

conda env list

Using Programs installed in a Conda Virtual Environment

In order to use the programs installed in a virtual environment the environment must be activate. To do this, first activate the base conda environment and then activate your desired virtual environment. This must be done whenever logging in to a new node, this process is also necessary in any slurm scripts which run programs using the environment.

Activate base environment.

source $HOME/miniforge/bin/activate

Your terminal prompt should now indicate that the base conda environment is active, e.g: (base) [username@computername ~]$

Activate an environment named 'myenv'

conda activate myenv

Your terminal prompt should then display the name of your active environment: (myenv) [username@computername ~]$

You can now use any programs installed to that virtual environment.