What is GNU Parallel?

GNU parallel is a shell tool for executing jobs in parallel using one or more compute nodes. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.

To setup required environment variables for Perl with base modules, please use following command:

System Version Command
HPC2021 20211222 module load gnuparallel/20211222

Sample SLURM batch script for GNU Parallel is located under /share1/gnuparallel/sample/ in HPC2021 system.

Additional Information

  1. GNU Parallel Tutorial
  2. GNU Parallel Cheat sheet
  3. GNU Parallel with examples in Bioinformatics