What is PGI Compiler?

The PGI Cluster Development Kit which enables use of clusters to tackle large scientific computing application is installed. This includes suite of optimizing compilers for Fortran 77(pgf77), Fortran 90(pgf90), Fortran (pgfortran), C(pgcc), C++ (pgc++) to build, debug and profile MPI and hybrid MPI/OpenMP HPC applications using major open source MPI implementations.

Using PGI Compilers

PGI Compilers & Tools have evolved into the NVIDIA HPC SDK in 2020. ITS has perpetual license of PGI Professional Edition. Users can keep using all previous releases in the HPC cluster systems.  Use of PGI compilers and tools are governed by the PGI End-User License Agreement(EULA). User must read and accept the PGI EULA before using the software at our HPC systems.

For HPC2015 system, user can set the required environment variables and path by loading the module.

PGI version MPI version module command
15.7 module load pgi/15.7
MPICH module load mpich/pgi/3.1.3
OpenMPI module load openmpi/pgi/1.8.4
MVAPICH module load mvapich/pgi/2.0
16.10 module load pgi/16.10
MVAPICH module load mvapich/pgi/2.1
17.7 module load pgi/17.7
MPICH module load mpich/pgi/3.2.0
OpenMPI module load openmpi/pgi/1.10.2
18.5 module load pgi/18.5
MPICH module load mpich/pgi/3.2.1
OpenMPI module load openmpi/pgi/2.1.2
MVAPICH module load mvapich/pgi/2.2
19.7 module load pgi/19.7
OpenMPI module load openmpi/pgi/3.2.1

Compilation commands

Serial C programs :
      pgcc [options] program.c -o program.exe
Serial Fortran 77 programs :
      pgf77 [options] program.f -o program.exe
Serial Fortran 90 programs :
      pgf90 [options] program.f90 -o program.exe

MPI C programs :
      pgcc -Mmpi=[mpiver] [options] program.c -o program.exe
MPI C++ programs :
      pgc++ -Mmpi=[mpiver] [options] program.cpp -o program.exe

MPI Fortran 77 or Fortran 90 programs in fixed form :
      pgf77/pgf90 -Mmpi=[mpiver] [options] program.f -o program.exe

MPI Fortran 77 or Fortran 90 programs in free form :
      pgf77/pgf90 -Mmpi=[mpiver] -Mfree [options] program.f -o program.exe

Examples of MPI program (greetings.c & helloworld.f90) are located in ~/pbs-sample/demo-MPI/ or ~/slurm-sample/demo-MPI/.

Using Compiler Options

Quick list of arguments for the compiler can be found using the compile command with argument -help .

Most commonly used arguments for the compilers are :

-fast Common optimizations
-Mfixed Assume fixed-format source for Fortran programs
-Mfree Assume free-format source for Fortran programs
-Mprof=lines,mpich Profiling a mpi program
-Mprof=lines Profiling a serial or multi-threaded program
-g Generates debug information for use with debuggers

To achieving peak performance, a list of tuning flags is shown below:

Option Flag Usage
-Mconcur Enable auto-parallelization; for use with multi-core or multi-processor targets.
-mp Enable OpenMP; enable user inserted parallel programming directives and pragmas.
-Mprefetch Control generation of prefetch instructions to improve memory performance in compute-intensive loops.
-Mmpi=mpiver Link with mpi libraries (e.g. -Mmpi=mpich)
-tp=target Optimized for target processor type:

Processor Type System Flag
Ivy Bridge HPC2015 (SP Node) -tp=ivybridge
Haswell HPC2015 (GP Node) -tp=haswell
Cascade Lake HPC2015 (mehpc2 node) -tp=skylake

The instruction set is upward compatible. Therefore, applications compiled with -tp=haswell can run on Haswell or Cascade Lake processors, but not on Ivy Bridge.

-Mscalapack Link with PGI precompiled ScaLAPACK libraries

For MVAPICH and OpenMPI, instead of PGI option -Mmpi, corresponding compiler wrappers should be used. Please visit ITS software guide of OpenMPI and MVAPICH for details.

Running Program

Test serial program
In order to test serial program, you can use this command : ./program.exe
The “./” tells the system to run the program in the current directory.

Test MPI program
Use this command to test MPI program in the master node : mpirun -np 10 ./program.exe
You can change “-np xxx” where xxx is the number of processes for testing.

Profiling a program

Profile a program means checking which line of code is the most time consuming. In order to profile a program, you should add the profile argument when you compiler the program. The profile argument are -Mprof=lines for serial programs or -Mprof=lines,mpich for MPI programs.

After the program is run, serial profile program would output a file called pgprof.out. You can display this file using the command pgprof .

For parallel profile program, after the program is run, there are 4 files created and they are called pgprof.out, pgprof1.out, pgprof2.out and pgprof3.out . Only the last 3 can be displayed. You can use this command to display the result :
pgprof -merge pgprof1.out pgprof2.out pgprof3.out
Detailed PGPROF profiler guide can be obtained at http://www.pgroup.com/doc/pgprofug.pdf.

Debugging program

PGDBG is a debugger for Fortran, C, C++ programs. It supports debugging of certain types of parallel applications:

  • Multi-threaded and OpenMP applications
  • MPI applications
  • Hybrid (multithreads/OpenMP + MPI) applications

To build a program for debug, compile with the -g option (this will sets the compiler optimization to level zero, i.e. no optimization).
Then you can invoke the debugger by:

For serial or multithreaded program
pgdbg ./program.exe
For MPI program (PGDBG can debug up to 256 MPI processes), 20 MPI processes in following example
pgdbg -mpi -n 20 ./program.exe
Detailed PGDBG debugger user guide can be obtained at  http://www.pgroup.com/doc/pgdbgug.pdf

Additional Information

PGI Official website: http://www.pgroup.com/
PGI CDK Cluster Development Kit: http://www.pgroup.com/products/pgicdk.htm
PGI Frequently Asked Questions: http://www.pgroup.com/support/faq.htm