TotalView

TotalView

What is TotalView?

TotalView from PERFORCE is a debugging tool particularly suitable for debugging of parallel applications. It provides both X Window-based Graphical User Interface (GUI) and command line interface (CLI) environments for debugging. This example illustrates how to use TotalView on the FAS Cannon cluster.

The specific example, wave_mpi.f90, solves the wave equation in parallel using MPI.

Contents:

wave_mpi.f90: Example MPI Fortran code

Using TotalView on Cannon:

To use Totalview on Cannon, first you need to load the Totalview module-file to set the correct environment settings. This is done most conveniently by placing the command module load totalview/2019.0.4-fasrc01 in your .bashrc startup file.

In order to debug MPI parallel applications, in addition to the TotalView software module, you need also to load appropriate Compiler and MPI modules, for instance:

module load totalview/2019.0.4-fasrc01
module load intel/21.2.0-fasrc01
module load openmpi/4.1.3-fasrc01

To use Totalview, your code must be compiled with the -g compile option. It is also recommended to suppress any level of optimization by compiling your application with the -O0 option.

Fortran 77: mpif77 -g -O0 -o tv_test.x tv_test.f
Fortran 90: mpif90 -g -O0 -o tv_test.x tv_test.f90
C:          mpicc -g -O0 -o tv_test.x tv_test.c
C++:        mpicxx -g -O0 -o tv_test.x tv_test.cpp

Note: The instrumented executable should be used for debugging only, not in production runs. After your code is up and running, for actual production runs you need to recompile your application with the desired level of optimization.

To use Totalview, you need to log in with an X window forwarding enabled. If you access Cannon from a Unix-like system, you have to use the -X or -Y option to ssh. The -Y option often works better for Mac OS X. For instructions on how to enable X11 forwarding when accessing Cannon from Windows click here.

ssh -l username login.rc.fas.harvard.edu -YC

Note: Alternatively, one may use the VDI portal and then launch a "Remote Desktop" app.

After loading the Totalview module and compiling with the -g option, request an interactive session:

salloc -p test --x11 -n 4 -t 00-02:00 --mem-per-cpu=4000

This will start an interactive (bash) shell and load the module-files included in your startup .bashrc file. Then launch the debugger with one of the following commands:

totalview mpirun -a -np 4 ./tv_test.x

or

mpirun -np 4 -tv ./tv_test.x

or

mpirun -np 4 -debug ./tv_test.x

The TotalView startup GUI will pop up and display debugging startup parameters, as illustrated below. After reviewing them, click OK.

Go to the process window, and click the "Go" button.

Note: At this stage Totalview displays the source code of the mpirun function, NOT the source code of your application.

After you click "GO" in the process window, a small window will pop up, asking whether the mpirun process should be stopped. Click "Yes".

Then, in the "Stack Trace" section of the process window you should see the name of the main program of your application. You can now display the source code by clicking on it. To start debugging, create a break point by clicking on a line number in the source pane, and click "Go". After that, you can use other buttons ("Next", "Step", "Out", etc).

References:

• Official Totalview Documentation
• Totalview tutorial, Lawrence Livermore National Laboratory
• Totalview tutorial - common functions, LLNL