Preparing for and installing MPE and JUMPSHOT.
MPE or multi-processing environment provides a complete set of performance analysis tools for MPI (Message Passing Interface) programs. It is based on a post processing approach. The tools are broadly divided into: a) Tracing Library b) Animation Libraries and c) Logging Libraries.
Tracing Libraries - These trace all MPI calls. Each MPI call is preceded by a line that contains the rank in MPI_COMM_WORLD of the calling process, and followed by another line indicating that the call has completed. Most send and receive routines also indicate the values of count, tag, and partner (destination for send, souce for receives). Output is to the standard output. Sometimes during the excution of a parallel program you need to ensure that only a few (often just one) processor at a time is doing something. The routine MPE_Seq_begin and MPE_Seq_end allow you to create a "sequential section" in a parallel program. As the MPI-2 standard makes it easy for users to define the routine to be called when an error is detected by MPI, the program can be made to automatically start a debugger when an error is detected, so that the problem can be diagnosed easily. Thus the call MPE_Errors_call_dbx_in_xterm allows you to just that. In addition, one can also compile the MPE library with debugging code included. The MPI profiling interface provides a convenient way to add performance analysis tools to any MPI implementation. Thus MPE uses may also build and use these libraries with any MPI implementation. The first profiling library is that which accumulates the time spent in MPI routines and forms part of this tracing library. The flag -mpitrace is need to compile and link any MPI program with the tracing libraries.
Animation Libraries - These are a simple form of real-time program animation that requires X window routines. The MPE graphics library contains routines that allow a set of processes to share an X display that is not associated with any one specific process. The flag -mpianim is required to compile and link any MPI program with animation libraries.
Logging Libraries - These are the most useful and widely used of all libraries in MPE. They form the basis to generate the log files from user MPI programs. The log file format we are concerned with is the SLOG format that is used by JUMPSHOT-3 a SLOG file viewer. By default the MPE enabled code will generate a CLOG format file. By running a program clog2slog the required format for jumpshot is got. In addition to using the predefined MPE logging libraries to log all MPI calls, MPE logging calls can be inserted into user's MPI program to define and log states. These states are called User-Defined states. States may be nested, allowing one to define a state describing a user routine that contains several MPI calls, and displays both the user-defined state and the MPI operations contained within it.
We discuss detailed examples of how to use these profiling libraries in Microway Working Note 16. However for visualising program execution at various nodes, optimization, debugging, and tracing times for the MPI calls and inter-node communication the use of MPE becomes evident. Further to compare different parallelization schemes (domain v/s functional decomposition for example), architectures (shared v/s distributed), interconnets (Myrinet v/s Dolphin Wulfkit) for a particular problem without too much overhead from the inherent codes the MPE libraries offer the ideal solution.
Preparing for and installing MPE and JUMPSHOT.
For the installation of MPE and jumpshot the Java 2 SDK (we used Java 2 v 1.3 with Red Hat Linux 7.0) must be installed and properly configured on each node in the cluster. The details are beyond the scope of this Working Note. Also a patch is needed if the command gcc --version returns 2.96. This patch is called 5597. We use mpich-1.2.1.
Next either rsh or ssh must be properly configured for the cluster of nodes. In our installation we use rsh, and have a single master and 3 nodes called node2 node3 and node4. For details of installing and configuring rsh refer to Microway Working Note 1.
Issue the following commands to install MPICH 1.2.1 assuming that Java 2 SDK is installed and working correctly on each node in the cluster.
>gunzip -c mpich-1.2.1.tar.gz | tar xovf -
>cd mpich-1.2.1
Copy the patch 5597 into the current directory.
>patch -p0 < 5597
Now you should see that six files are patched. Note if you do not see this DO NOT reapply the patch. Issue cd .. and rm -rf mpich-1.2.1 and go back an gunzip and restart.
After the patch is successfully set issue in the mpich-1.2.1 directory the following commands.
>./configure --with-arch=LINUX -comm=shared
(We are using a standard ch_p4 device - 100Mbs Intel-Ether Express Pro ethernet cards. The -comm= shared flag is needed if the node has more than one processor (SMP). If no other options are used ROMIO I/O, MPE and JUMPSHOT with UPSHOT are all installed by default. Use the -noromio option if you do not want ROMIO I/O, --without mpe for not installing MPE, --without jumpshot for not installing jumpshot and --without upshot for not installing upshot.)
Then type make and this completes the installation.
Make sure that the PATH in the root and user .bash_profile are set to indicate the following directories:
.../mpich-1.2.1/bin (for mpirun ... ...etc)
.../mpich-1.2.1/mpe/bin (for clog2slog ... ...etc)
.../mpich-1.2.1/mpe/viewers/jumpshot-3/bin (for jumpshot ... ...etc)
Edit the machines.LINUX file in .../mpich-1.2.1/util/machines directory to reflect the addresses of the nodes and the number of processors in each node. Ours looks as shown below:
This is very important as all types of conflict errors will be generated if mpirun is used with this step not being done correctly. In particular this is true for SMP nodes.
On reboot the command from a GNOME xterm window >jumpshot should be as shown below.
Next in the folder .../mpich-1.2.1/mpe/contrib/mandel issue make. Next issue ./pmandel and you should see a screen as shown below. Remember to set xhost + master, etc for all the nodes so that Xwindows can communicate.
In the directory .../mpich-1.2.1/examples type make. There are number of tests in the test/ sub-directory that can now be run. In the basic subdirectory is a program called cpilog that we run as shown below. A .clog file is created that is converted to .slog using clog2slog and is viewed with jumpshot. This is shown below.
Typing jumpshot cpilog.slog will open the window shown below.
Selecting Display in the above figure enables viewing both MPI-Process and Thread. This is shown below.
By selecting Change Orientation and say Broadcast for example one can analyze in detail the program behavior as shown below.
This completes the installation and running of MPE and Jumpshot.
BACK TO MICROWAY APPLICATION NOTE INDEX