Debugging MPI applications is notoriously hard. The Intel Trace Analyzer & Collector (ITAC) can be used to generate a trace while running an application, and visualizing it later for analysis.


You will need an MPI program (C/C++ or Fortran) to instrument and run.

Step by step

The following steps are the easiest way to use the Intel Trace Analyzer, however, more sophisticated options are available.

  1. Load the relevant modules. The exact modules may differ from system to system, but will typically include the itac module and a compatible Intle toolchain, e.g.,
    $ module load intel/2015a
    $ module load itac/
  2. Compile your application so that it can generate a trace:
    $ mpiicc -trace myapp.c -o myapp
    where myapp.c is your C/C++ source code. For a Fortran program, this would be:
    $ mpiifort -trace myapp.f -o myapp
  3. Run your application using a PBS script such as this one:
    #!/bin/bash -l
    #PBS -N myapp-job
    #PBS -l walltime=00:05:00
    #PBS -l nodes=4
    module load intel/2015a
    module load itac/
    # Set environment variables for ITAC.
    # Unfortunately, the name of the script differs between versions of ITAC
    source $EBROOTITAC/bin/itacvars.sh
    mpirun -trace myapp
  4. When the job is finished, check whether files with names myapp.stf.* have been generated, if so, start the visual analyzer using:
    $ traceanalyzer myapp.stf

Further information

Intel provides product documentation for ITAC.