Skip to content

Running Python on ARGO

Running Serial Python Jobs

To use the various version of Python on ARGO, you have to load the module Python. Currently the most up-to-date Python version is 3.8.4. Make sure you check all the available R modules installed on the cluster before submitting your script. Ensure that you specify the R module version you want to use.

If you type module show python/3.8.4, you will see the following:

-------------------------------------------------------------------
/cm/shared/modulefiles/python/3.8.4:

module-whatis    Adds the Python 3.8.4 binaries to your path.
setenv           OMP_NUM_THREADS 1
setenv           OPENBLAS_NUM_THREADS 1
setenv           MKL_NUM_THREADS 1
setenv           VECLIB_MAXIMUM_THREADS 1
setenv           NUMEXPR_NUM_THREADS 1
prepend-path     PATH /cm/shared/apps/python/3.8.4/bin
prepend-path     LD_LIBRARY_PATH /cm/shared/apps/python/3.8.4/lib:/cm/shared/apps/tcltk/8.6.8/lib
prepend-path     LIBPATH /cm/shared/apps/python/3.8.4/lib
prepend-path     CPATH /cm/shared/apps/python/3.8.4/include
prepend-path     PKG_CONFIG_PATH /cm/shared/apps/python/3.8.4/lib/pkgconfig
-------------------------------------------------------------------

You can submit batch R jobs with a Slurm submission script. At the end of your Slurm script, you can run your python script with the following command:

python  <your-R-script>.py 

Python virtual environments

To have better control over the environment and packages you want to use for your Python jobs, it is recommended to use python virtual environments. See this page for more details:

Intel Python

Intel's distribution of optimized python is available on ARGO. Intel has optimized the most commonly used python packages, i.e., Numpy, Scipy, Scikit-learn using Intel MKL (Math Kernel Library) and Intel Data Analytics Acceleration (DAAL) libraries. On Intel architecture these optimized packages are expected to provide a significant performance boost. To make use of Intel's distribution of packages users need to load the Intel module 'intel/python/2018-10-p36' for Python version 3.6 or 'intel/python/2018-10-p27' for Python version 2.7.

  $ module load intel/python/2018-10-p36
  $ pip freeze
    daal==2019.0
    icc-rt==2019.0
    impi==2019.0
    intel-numpy==1.15.1
    intel-openmp==2019.0
    intel-scikit-learn==0.19.2
    intel-scipy==1.1.0
    .....

In order to check whether the intel optimized version of numpy and other packages has been loaded, the configuration of the packages can be checked as follows,

$ python3
Python 3.6.4 (default, Jun  7 2018, 10:05:32)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
mkl_info:
   libraries = ['mkl_rt', 'pthread']
   library_dirs = ['/opt/anaconda1anaconda2anaconda3/lib']
   define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
   include_dirs = ['/opt/anaconda1anaconda2anaconda3/include']
   .....

Optimized Tensorflow and Caffe

Intel-optimized Tensorflow and Caffe are also available on ARGO. Users are encouraged to use these optimized versions if the code is written for cpus only.