Software

On HPC:

1. The Operating System is CentOS 8.2. Windows / Mac software is not supported.

For running Jobs: Please refer to the Jobs Management section.

Important

If you have a job with independent computations, then Job arrays and Parallel Job Array are one of the most easiest ways to parallelize your computations. Follow the corresponding highlighted links for a much more detailed example.

Software Modules and environment

• Software Modules

HPC Miniconda

We have a centralized installation of Miniconda, TensorFlow and PyTorch in the HPC. Please refer to the highlighted sections for more details.

• Miniconda on the HPC
  • One Time Setup
  • Application Installation
  • Managing Environments
  • Migrating / Sharing Environment
  • Submitting Job Scripts
• Conda Environment Migration from $SCRATCH
  • conda-migrate
  • Description
  • Options
  • Usage
  • Finding the Conda Environment directories
  • Migrating Environments
  • Migrate All Valid Environments in a Directory
  • (Optional) Job Submission for Migrating Environments
• PyTorch on HPC
  • How to clone the PyTorch environment
  • Submitting Job Scripts
• TensorFlow (Keras) on HPC
  • How to clone the TensorFlow environment
  • Submitting Job Scripts

Matlab

• How to use MATLAB Parallel Toolbox
  • Running a Serial MATLAB Job
  • Running a Multi-threaded MATLAB Job with the Parallel Computing Toolbox
  • Overriding the 12 core limit
  • How Do I Know If My MATLAB Code is Parallelized?
  • Running Matlab on GPUs

Singularity

• Singularity Containers
  • What is a Container?
  • How do containers differ from virtual machines (VMs)
  • Docker
  • Singularity
  • Steps to use a Container
  • Using docker and singularity images from existing container libraries
  • Creating Singularity containers
  • Singularity Containers from scratch
  • Additional Documents
• Singularity - How to Run on HPC
  • Commands at a glance
  • Interact With Container
  • Files in a container
  • Singularity Overlays
  • GPU in a container
  • Sample job script
  • Build and Modify your own containers
  • Additional Documents
• Singularity Overlays
  • Creating an Overlay
  • Mounting an Overlay
  • Running with an Overlay
  • Analogy: Understanding Overlays
  • Sharing the Overlay
  • Job Submission

Miscellaneous

• R in HPC Miniconda
  • How to clone the R environment
  • Finding the Conda complementary packages (Tidyverse, Rstan etc)
  • How to install the Conda complementary packages
  • Submitting Job Scripts