What is PGI Compiler?
The PGI Cluster Development Kit which enables use of clusters to tackle large scientific computing application is installed. This includes suite of optimizing compilers for Fortran 77(pgf77), Fortran 90(pgf90), Fortran (pgfortran), C(pgcc), C++ (pgc++) to build, debug and profile MPI and hybrid MPI/OpenMP HPC applications using major open source MPI implementations.
Using PGI Compilers
PGI Compilers & Tools have evolved into the NVIDIA HPC SDK in 2020. ITS has perpetual license of PGI Professional Edition. Users can keep using all previous releases in the HPC cluster systems. Use of PGI compilers and tools are governed by the PGI End-User License Agreement(EULA). User must read and accept the PGI EULA before using the software at our HPC systems.
For HPC2015 system, user can set the required environment variables and path by loading the module.
| PGI version | MPI version | module command |
|---|---|---|
| 15.7 | — | module load pgi/15.7 |
| MPICH | module load mpich/pgi/3.1.3 | |
| OpenMPI | module load openmpi/pgi/1.8.4 | |
| MVAPICH | module load mvapich/pgi/2.0 | |
| 16.10 | — | module load pgi/16.10 |
| MVAPICH | module load mvapich/pgi/2.1 | |
| 17.7 | — | module load pgi/17.7 |
| MPICH | module load mpich/pgi/3.2.0 | |
| OpenMPI | module load openmpi/pgi/1.10.2 | |
| 18.5 | — | module load pgi/18.5 |
| MPICH | module load mpich/pgi/3.2.1 | |
| OpenMPI | module load openmpi/pgi/2.1.2 | |
| MVAPICH | module load mvapich/pgi/2.2 | |
| 19.7 | — | module load pgi/19.7 |
| OpenMPI | module load openmpi/pgi/3.2.1 |
Compilation commands
Serial C programs :
pgcc [options] program.c -o program.exe
Serial Fortran 77 programs :
pgf77 [options] program.f -o program.exe
Serial Fortran 90 programs :
pgf90 [options] program.f90 -o program.exe
MPI C programs :
pgcc -Mmpi=[mpiver] [options] program.c -o program.exe
MPI C++ programs :
pgc++ -Mmpi=[mpiver] [options] program.cpp -o program.exe
MPI Fortran 77 or Fortran 90 programs in fixed form :
pgf77/pgf90 -Mmpi=[mpiver] [options] program.f -o program.exe
MPI Fortran 77 or Fortran 90 programs in free form :
pgf77/pgf90 -Mmpi=[mpiver] -Mfree [options] program.f -o program.exe
Examples of MPI program (greetings.c & helloworld.f90) are located in ~/pbs-sample/demo-MPI/ or ~/slurm-sample/demo-MPI/.
Using Compiler Options
Quick list of arguments for the compiler can be found using the compile command with argument -help .
Most commonly used arguments for the compilers are :
| -fast | Common optimizations |
| -Mfixed | Assume fixed-format source for Fortran programs |
| -Mfree | Assume free-format source for Fortran programs |
| -Mprof=lines,mpich | Profiling a mpi program |
| -Mprof=lines | Profiling a serial or multi-threaded program |
| -g | Generates debug information for use with debuggers |
To achieving peak performance, a list of tuning flags is shown below:
| Option Flag | Usage | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| -Mconcur | Enable auto-parallelization; for use with multi-core or multi-processor targets. | ||||||||||||
| -mp | Enable OpenMP; enable user inserted parallel programming directives and pragmas. | ||||||||||||
| -Mprefetch | Control generation of prefetch instructions to improve memory performance in compute-intensive loops. | ||||||||||||
| -Mmpi=mpiver | Link with mpi libraries (e.g. -Mmpi=mpich) | ||||||||||||
| -tp=target | Optimized for target processor type:
The instruction set is upward compatible. Therefore, applications compiled with -tp=haswell can run on Haswell or Cascade Lake processors, but not on Ivy Bridge. |
||||||||||||
| -Mscalapack | Link with PGI precompiled ScaLAPACK libraries |
For MVAPICH and OpenMPI, instead of PGI option -Mmpi, corresponding compiler wrappers should be used. Please visit ITS software guide of OpenMPI and MVAPICH for details.
Running Program
Test serial program
In order to test serial program, you can use this command : ./program.exe
The “./” tells the system to run the program in the current directory.
Test MPI program
Use this command to test MPI program in the master node : mpirun -np 10 ./program.exe
You can change “-np xxx” where xxx is the number of processes for testing.
Profiling a program
Profile a program means checking which line of code is the most time consuming. In order to profile a program, you should add the profile argument when you compiler the program. The profile argument are -Mprof=lines for serial programs or -Mprof=lines,mpich for MPI programs.
After the program is run, serial profile program would output a file called pgprof.out. You can display this file using the command pgprof .
For parallel profile program, after the program is run, there are 4 files created and they are called pgprof.out, pgprof1.out, pgprof2.out and pgprof3.out . Only the last 3 can be displayed. You can use this command to display the result :
pgprof -merge pgprof1.out pgprof2.out pgprof3.out
Detailed PGPROF profiler guide can be obtained at http://www.pgroup.com/doc/pgprofug.pdf.
Debugging program
PGDBG is a debugger for Fortran, C, C++ programs. It supports debugging of certain types of parallel applications:
- Multi-threaded and OpenMP applications
- MPI applications
- Hybrid (multithreads/OpenMP + MPI) applications
To build a program for debug, compile with the -g option (this will sets the compiler optimization to level zero, i.e. no optimization).
Then you can invoke the debugger by:
For serial or multithreaded program
pgdbg ./program.exe
For MPI program (PGDBG can debug up to 256 MPI processes), 20 MPI processes in following example
pgdbg -mpi -n 20 ./program.exe
Detailed PGDBG debugger user guide can be obtained at http://www.pgroup.com/doc/pgdbgug.pdf
Additional Information
PGI Official website: http://www.pgroup.com/
PGI CDK Cluster Development Kit: http://www.pgroup.com/products/pgicdk.htm
PGI Frequently Asked Questions: http://www.pgroup.com/support/faq.htm