Debugging MPI applications is notoriously hard. The Intel Trace Analyzer & Collector (ITAC) can be used to generate a trace while running an application, and visualizing it later for analysis.
You will need an MPI program (C/C++ or Fortran) to instrument and run.
Step by step
The following steps are the easiest way to use the Intel Trace Analyzer, however, more sophisticated options are available.
Load the relevant modules. The exact modules may differ from system to system, but will typically include the itac module and a compatible Intle toolchain, e.g.,
$ module load intel/2015a $ module load itac/9.0.2.045
Compile your application so that it can generate a trace:
$ mpiicc -trace myapp.c -o myappwhere myapp.c is your C/C++ source code. For a Fortran program, this would be:
$ mpiifort -trace myapp.f -o myapp
Run your application using a PBS script such as this one:
#!/bin/bash -l #PBS -N myapp-job #PBS -l walltime=00:05:00 #PBS -l nodes=4 module load intel/2015a module load itac/9.0.2.045 # Set environment variables for ITAC. # Unfortunately, the name of the script differs between versions of ITAC source $EBROOTITAC/bin/itacvars.sh cd $PBS_O_WORKDIR mpirun -trace myapp
When the job is finished, check whether files with names myapp.stf.* have been generated, if so, start the visual analyzer using:
$ traceanalyzer myapp.stf
Intel provides product documentation for ITAC.