Skip to content

Installing User Packages in ARGO

Installing Packages

The main method of installing custom packages in ARGO is via the "configure--make--make-install" combo. For this, you first have to find a source distribution of the package you want to install. Then, either use the ./configure script that came with your distribution or manually edit the makefiles to set where you want to install the package inside your home directory. If the distribution came with a ./configure script, then you can specify the install path by adding the prefix option as shown below:

$ ./configure prefix=<install-location>

Using Source Install

Here we provide an example of how to install a custom package using a source distribution. We are using the gnu common lisp source package for demonstration purposes. You can download the latest source files from here: http://gnu.mirror.vexxhost.com/gcl/. You may need to review the instructions on Uploading Data in order to get the tarball to ARGO. Then unpack the contents of the tarball somewhere inside your ARGO home directory with one of the following commands, depending on the package:

$ tar xvfz <package-tarball>.tgz
 - or -
$ tar xvfz <package-tarball>.tar.gz

And then cd into the directory that was created:

$ cd <the-new-tarball-directory>  # probably "gcl" for our example

Let's assume that you want to install this package in the following directory: 

/home/$USER/packages/gcl-<version.number>.
Make sure that directory has been created: 

$ mkdir -p /home/$USER/packages/gcl-<version.number>
Then from the directory where you had unpacked the contents of the tarball, run the following three commands. Note that each of these commands is likely to generate a large amount of text. Keep you eyes on the output to make sure there are no errors. 

$ ./configure prefix=/home/$USER/packages/gcl-<version.number>
$ make
$ make install
If the compilation goes through without any errors, you will find the gcl executable inside the /home/$USER/packages/gcl-<version.number>/bin/ directory. Now the only thing left to do is to setup the appropriate environment parameters such that the system recognizes the new gcl command. We do this by creating a custom module file which will contain the path information of the gnu common lisp executable file in your home directory. We need to load it before we want to use the gcl command. We shall go over how to create and load custom module files in the Creating and Loading Custom Modules section below. At this point you can safely remove the tarball, and the directory it created.

Installing Python Packages

We recommend using a Python Virtual Environment for installing Python packages on the ARGO cluster.

Creating and Loading Custom Modules

You can use environment modules to manage software you have installed in your home directory by creating your own modules to load the software installed in your home directory. To load the modules you created, you have to first load the use.own module and then the module created by you.

$module show use.own
 -------------------------------------------------------------------
 /cm/local/modulefiles/use.own:

 module-whatis   adds your own modulefiles directory to MODULEPATH 
 module          use --append /home/your-user-id/privatemodules 
 -------------------------------------------------------------------

You need to create a privatemodules (if it does not exists) directory, cd to privatemodules. Add module files in this directory and when creating a module file, name the file according to the software because the module will show up under this given name. Given below is an example of a module file which can be used to load the gnu common lisp we installed earlier.

#%Module 1.0
#
# Sets up variables for my local install of GCL, for use with the
# 'environment-modules' package.
#
proc ModulesHelp { } {
    puts stderr "\tAdds local install of GCL to your environment variables,"
}

module-whatis "adds local install of GCL  to your environment variables"

# You can specify additional modules (if any) to be loaded before loading your
# custom module like shown below.
# Note: for python we don't need any additional module to be loaded beforehand.
#if ![is-loaded <some-required-module-name>] {
#       module load <some-required-module-name>
#}

# Add in required module(s) needed as prerequisite.
# We don't need any prerequisite module for using GCL
#prereq <module_name>

# Add the path to the GCL binaries
prepend-path PATH /home/<userID>/packages/gcl-<version.number>/bin/

## Add the path to the GCL libraries
prepend-path LD_LIBRARY_PATH /home/<userID>/packages/gcl-<version.number>/lib/
Name this file as gcl-<version.number> and save it in your privatemodules directory. To load your custom module use: 

$ module load use.own
Now if you do: 

$ module avail
You will see your module being shown in the bottom of the list under custom modules. Now load your module just like any other. 
$ module load gcl-<version.number>

The above commands can be embedded inside your grid engine job script in the same way when you load public modules. Note: You must always load the use.own module before using your custom module in the slurm job submission script.

Creating Module Files for Python Packages

In addition to the parameters shown in the above simple module file, if you are creating module files for python packages (such as hcluster) we must also specify the relevant PYTHONPATH environment as shown below:

#%Module 1.0
...
if ![is-loaded python/python2.7] {
    module add python/python2.7
}
if ![is-loaded numpy/openblas/1.9.2] {
    module add numpy/openblas/1.9.2
}

# Specify here the python libraries needed before using your library (in
# this case hcluster)
prereq python/python2.7
prereq numpy/openblas/1.9.2
...
# Add the path where we have installed hcluster to the PYTHONPATH
# environment variable
prepend-path PYTHONPATH /home/<your-userID>/bin

# Since hcluster is a library we also add the path where the .so file is
# located to the system LD_LIBRARY_PATH
prepend-path LD_LIBRARY_PATH /home/<your-usrID>/bin/lib/

Note that you may need to add more PATH parameters depending on what package you are creating the module for. Consult your package documentation for details about its dependencies, and based on that see if you require additional parameters in your module file.

Reference

1.The man-page for creating modules can be found here:[1]