Visualising MPI Communication With MPE

Visualising with MPE

A couple of weeks ago I had the task of accelerating some fluid dynamics code using MPI (message passing interface) which uses messages (as you might have guessed) to help parallelise code (versus OpenMP which uses a shared memory model). This was done across four nodes each containing two dual core CPUs (so 8 nodes) and achieved a speedup of 11x which I was fairly pleased with.

The most interesting part of the project was visualising the communication between the nodes. To do this myself would have been nigh-on impossible: dealing with disparate nodes across a large system each with their own timings it would be impossible to synchronise the communications and logging correctly to collect and visualise later.

Fortunately for me there exists the MPE project which supplies performance visualisations for MPI.

Once having configured MPE successfully on the cluster, with MPE is was able to add a couple of lines to my code (see below) to tell it that I wanted a visualisation.

There are two ways of doing this: the first is to have MPE log the images to disk for you to assemble later (using xwd) whilst the second way is to have it spawn a window to show you it in real time. Since this simulation would require around 10,000 simulations and I didn't fancy stitching together so many images, I decided to use the second option: spawn a window via X-forwarding and do a screen cap. Yes, hacky and horrible I know!

Anyway, here is the outline of the code I had to add to get it up and running. As you can see it's all fairly simple and doesn't interfere with any of the pre-existing MPI code. Hats off to ANL for this, it's very useful.

#include "mpe.h"
#include "mpe_graphics.h"

//...the start of your program here and initialise your MPI env. as per usual

MPE_XGraph graph;

// Open the MPE graphics window of size 400x400 at  600,-1
int ierr = MPE_Open_graphics( &graph, MPI_COMM_WORLD,NULL,
                              600, -1, 400, 400, 0 );

// Alternatively you can set the capture file...
//ierr = MPE_CaptureFile(graph, "outputimage", 1);

if(ierr != MPE_SUCCESS)
{
    printf("Error Launching X world\n");
    MPI_Abort(MPI_COMM_WORLD, 1);
    exit(1);
}

//...skip out some code here and we enter the main body of our MPI code.

MPI_Isend(&cells[ topload    ], NX, structTypeTop, (rank+1) % size,       1, MPI_COMM_WORLD, &request);
MPI_Isend(&cells[ bottomload ], NX, structTypeBot, (rank+size-1) % size,  2, MPI_COMM_WORLD, &request);        

// This forces our graph to update after having seen the most recent comms
ierr = MPE_Update( graph );    
MPI_Recv(&cells [ bottomloadR], NX, structTypeTop, (rank+size-1) %size,   1, MPI_COMM_WORLD, &status);
MPI_Recv(&cells [ toploadR   ], NX, structTypeBot, (rank+1)%size,         2, MPI_COMM_WORLD, &status);
// Again, after receiving let's update our graph
ierr = MPE_Update( graph ); 

// ... continue looping round. When we are finished with MPI let's sync our clocks..
MPE_Log_sync_clocks();
MPE_Finish_log(argv[0]);
// And also close our graphics
MPE_Close_graphics(&graph);