Using MPI sockets

This article will present how to use MPI to create a remote socket and use it through MPI calls. Remember that we have the profiler - the library part that uses the profiling interface of MPI to profile the program - and the display - that displays the information sent by the profiler - parts that communicate.

First of all a research was made in order to try to find out how to create a socket with MPI on the profiler and communicate with some other socket library on the display. So far no example were found using that approach, and as this is a technical test, no real implementation was done that way.

The approach used here is to bind the profiler and display communicators using a technique similar to MPI_Spawn but that doesn't require the 2 softwares to be tight together. This is done using the MPI_Open_port functions.

The code wasn't modifier a lot from the MPI Spawn approach, as you are going to see. The reference used to understand and develop that approach was actually the MPI standard website: 5.4.6. Client/Server Examples

The profiler side - server side

The global idea of that approach is for the profiler to open a port, and wait for some display to connect on it. The idea can be pushed further, if needed, to allow several display to connect on a single profiler (sharing the view of the program on several display for example).

Actually what was modified from the Spawn example is the way to connect the profiler and the display together. Rather than calling MPI_Spawn, MPI_Open_port was used, and few lines were added just before finalizing the execution.

Opening the port

int start_child(char* command, char* argv[])
{
  MPI_Open_port(MPI_INFO_NULL, port_name);

  /* child doesn't find it...
    sprintf(published, "%s-%d\0", PROFNAME, world_rank);

    MPI_Publish_name(published, MPI_INFO_NULL, port_name);*/

  fprintf(stderr, "!profiler!(%d) open port '%s'\n", world_rank, port_name);

  fprintf(stderr, "!profiler!(%d) waiting for a child...\n", world_rank);

  MPI_Comm_accept(port_name, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm);

  fprintf(stderr, "!profiler!(%d) got a child!\n", world_rank);

  int r;
  MPI_Comm_rank(intercomm, &r);
  fprintf(stderr, "!profiler!(%d) is %d on parent!\n", world_rank, r);

  // wait for a message that "I'm dying"
  if ( PMPI_Irecv(&(quitmessage[0]), INTRA_MESSAGE_SIZE, INTRA_MESSAGE_MPITYPE, CHILD_RANK, INTERCOMM_TAG, intercomm, &dead_child) != MPI_SUCCESS )
    {
      intercomm = MPI_COMM_NULL;

      fprintf(stderr, "!profiler!(%d) communication failed!\n", world_rank);
      intercomm = MPI_COMM_NULL;
      return FAILURE;
    }

  char mess[INTRA_MESSAGE_SIZE];
  sprintf(mess, "%d IsYourFather\0", world_rank);


  sendto_child(mess);

  PMPI_Barrier(MPI_COMM_WORLD);


  return SUCCESS;
}

Finalizing the communication

int wait_child(char* mess)
{
  // send my death
  if ( sendto_child(mess) == SUCCESS )
    {
      // wait his death
      if ( PMPI_Wait(&dead_child, MPI_STATUS_IGNORE) == MPI_SUCCESS )
        {
          fprintf(stderr, "!profiler!(%d) received its child death!\n", world_rank);
          //MPI_Unpublish_name(published, MPI_INFO_NULL, port_name);
          MPI_Close_port(port_name);
          return SUCCESS;
        }
    }

  return FAILURE;
}

The display side - the client side

On the display side, the same kind of modification had to be done. Rather that using information from the father's communicator, a connection to a port is performed.

The MPIWatch::getWatcher method

MPIWatch* MPIWatch::getWatcher(char port_name[])
{
    if ( instance == 0 )
    {
        MPI::Init();

        std::cout << "Try to connect to " << port_name << std::endl;

        parent = MPI::COMM_WORLD.Connect(port_name, MPI::INFO_NULL, 0);

        if ( parent == MPI::COMM_NULL )
        {
            std::cerr << "Cannot connect with the parent program! Aborting." << std::endl;
            MPI::Finalize();
            return 0;
        }

        std::cout << "Connection with parent completed!" << std::endl;

        instance = new MPIWatch();
    }

    return instance;
}

Running it!

The main difference here is that on the previous version the display was starting by itself. Now it has to be started separately, and actually one per MPI process. Some attempts were made to use the name publication described in the standard (see the reference further up) but for a unknown reason the display part never found the profiler published name.So far, 1 port is open per MPI process - or 1 name was published - and each display connect on 1 of them through command line input.

Console 1: run MPI

$> mpiexec -n 2 mpi_ring
!profiler!(0) open port '3449421824.0;tcp://192.168.0.2:48251+3449421825.0;tcp://192.168.0.2:36965:300'
!profiler!(1) open port '3449421824.0;tcp://192.168.0.2:48251+3449421825.1;tcp://192.168.0.2:52304:300'

Console 2-3: run the display

$> mpidisplay '3449421824.0;tcp://192.168.0.2:48251+3449421825.0;tcp://192.168.0.2:36965:300'

$> mpidisplay '3449421824.0;tcp://192.168.0.2:48251+3449421825.1;tcp://192.168.0.2:52304:300'

The current implementation is a little more complicated to run than the spawn version, but doesn't have any error code when finishing. It also allows more flexibility in the future, to allow more than one display on a single profiler, and any other idea that requires a more flexible approach than a spawn process (like been able to connect a display in the middle of a run and disconnect at will, to see if the program is deadlock etc).

Limitations

The port information are rather long and it is quite not user friendly to have to lookup the profiler output and copy/paste the port information into the display. Further investigation have to be made on that part, in order to either manage to find the name publication problem, or to find a way to look for the port with a more automatic fashion.The actual name publication idea was to publish a name, like 'profiler-mpirank' to look up for - or with any string given by the user instead of profiler. This will allow the display to be started in a single command, that will only need to know 2 information: the base name of the profiler and the number of MPI process to connect to!

The other limitation is not a real one, but more like a bug on the current implementation. A barrier was added to wait for every MPI process to get a display, and isn't that much of a problem, as no high performance are required for that project. The problem arises when one display is closed while the program is running. The current implementation doesn't catch it, and deadlocks. Further investigation will obviously be done on that problem later on.

Source code

As for the previous version the source code is available on http://www.megaupload.com/?d=ZXJGHBPQ. It is a test version, not very clean, and buggy (as explained above). Later on a post will be done on how to use the library with a MPI code in C.

Further work

The preliminary technical overview of the project is about to be over. Now that the basis of the project techniques are setted up, are more detailed reasoning will be done on the project functional requirements. As part of the Project Preparation course of the MSc, some risk analysis and workplan for the overall project has to be done as well and will be published here as well.

Using MPI_Spawn

This article will present how to use MPI_Spwan and what are the problem associated with it. This will first show the profiler code, then the display code. And finally discuss the problems.

Spawn the interface: profiler point of view

In order to spawn the interface, the PATH variable was exported in order to contain the path to the executable mpidisplay, that is the simple interface developed for this test. It is basically counting the number of calls to some of the communication function of MPI.

The spawning actually occurs in the MPI_Init overloaded function :

int world_rank;
MPI_Comm intercomm = MPI_COMM_NULL;
int intercomm_child_rank = 0; 

int MPI_Init(int* argc, char ***argv)
{
  int ret;

  ret = PMPI_Init(argc, argv);

  PMPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

  fprintf(stderr, "!profiler(%d)! MPI_Init()\n", world_rank);
  
  // spawn the interface
  MPI_Comm_spawn("mpidisplay", MPI_ARGV_NULL, 1, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm, MPI_ERRCODES_IGNORE);

  return ret;
}

This is simply starting the display when the profiler is started through mpiexec and link them together. But as soon as MPI_Finalize is called, both of them are killed, and the interface is closed. Thus a trick was used to make the profiler waiting for the child to be closed to stop running.

The idea is that the display sends a message to the profiler when it is closed, and that the profiler waits on this message with an asynchronous receive from the beginning. When MPI_Finalize is called on the profiler, a MPI_Wait of that message is performed, basically waiting for the display to be closed to resume. The profiler also send information about is imminent death, to display the information if needed on the display.

#define CHILD "mpidisplay"
#define CHILD_ARGS MPI_ARGV_NULL

int world_rank;
MPI_Comm intercomm = MPI_COMM_NULL;
int intercomm_child_rank = 0; 

static Intra_message quitmessage[INTRA_MESSAGE_SIZE]; 
MPI_Request dead_child = MPI_REQUEST_NULL;

int MPI_Init(int* argc, char ***argv)
{
  int ret;
  Intra_message message[INTRA_MESSAGE_SIZE];

  ret = PMPI_Init(argc, argv);

  PMPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
  //PMPI_Comm_size(MPI_COMM_WORLD, &world_size);

  fprintf(stderr, "!profiler(%d)! MPI_Init()\n", world_rank);

  
  // spawn the interface
  MPI_Comm_spawn(CHILD, CHILD_ARGS, 1, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm, MPI_ERRCODES_IGNORE);

  sprintf(message, "%d Init\0", world_rank);

  PMPI_Ssend(message, INTRA_MESSAGE_SIZE, INTRA_MESSAGE_MPITYPE, intercomm_child_rank, 0, intercomm);

  // wait for a message that "I'm dying"
  PMPI_Irecv(&(quitmessage[0]), INTRA_MESSAGE_SIZE, INTRA_MESSAGE_MPITYPE, intercomm_child_rank, 0, intercomm, &dead_child);// check each time if the child is dead...

  return ret;
}

int MPI_Finalize(void)
{
  int ret;

  fprintf(stderr, "!profiler!(%d): MPI_Finalize()\n", world_rank);

  if ( dead_child != MPI_REQUEST_NULL )
    {
      Intra_message message[INTRA_MESSAGE_SIZE];

      sprintf(message, "%d Finalize\0", world_rank);

      // send my death to the display
      PMPI_Ssend(message, INTRA_MESSAGE_SIZE, INTRA_MESSAGE_MPITYPE, intercomm_child_rank, 0, intercomm);

      fprintf(stderr, "!profiler!%d is waiting for its child...\n", world_rank);

      // wait for the display to quit
      PMPI_Wait(&dead_child, MPI_STATUS_IGNORE);
      fprintf(stderr, "!profiler!%d finished waiting...\n", world_rank);
    }

  ret = PMPI_Finalize();

  return ret;
}

Spawn the interface: display point of view

The display was implemented using Qt, and is therefore in C++. The MPI calls are the same, just organized in a Object Oriented fashion.

When the child is spawned, it can retrieve its parent information, and do so in order to get the special communicator. Then it simply uses normal MPI communication with it.

The MPIWatcher class was written to handle the MPI communication. It is implementing the singleton design pattern. The MPI init code are therefore present in the global call that creates the object, and are normally performed only once (as the object is carried by until the end of the program).

MPIWatch* MPIWatch::getWatcher(void)
{
    if ( instance == 0 )
    {
        //MPI::Intercomm parent = MPI::COMM_NULL;
        int parentSize;

        MPI::Init();
        parent = MPI::Comm::Get_parent();

        if ( parent == MPI::COMM_NULL )
        {
            std::cerr << "Cannot connect with the parent program! Aborting." << std::endl;
            //parent.Abort(-1);
            MPI::Finalize();
            return 0;
        }

        parentSize = parent.Get_remote_size();

        if ( parentSize != 1 )
        {
            std::cerr << "Parent communicator size is " << parentSize << "! It should be 1. Aborting." << std::endl;
            parent.Abort(-1);
            return 0;
        }

        instance = new MPIWatch();
    }

    return instance;
}

The instance process to catch up message will be discuss later. Basically the MPIWatch do synchronized receives from his father, and push the result on a stack, that is read by the interface.

When the window is closed, the MPIWatch object has to be destroyed, and the actual message is therefore sent to the father.

bool MPIWatch::delWatcher()
{
    if ( ! instance )
        return false;

    if ( instance->isRunning() )
        return false;

     QString s(MESSAGE_QUIT);

     parent.Ssend(s.toStdString().c_str(), INTRA_MESSAGE_SIZE, INTRA_MESSAGE_MPITYPE, 0, 0);

     MPI::Finalize();
     parent = MPI::COMM_NULL;

    delete instance;
    instance = 0;

    return true;
}

Problems with spawned instances

The major issue with the spawn interface is the actual call to MPI_Finalize. When one of the child or parent calls it, the ORTE process - the daemon that handles the MPI communication on OpenMPI and MPICH should have something similar - kills the other. Therefore even with the trick to wait for the display from the profiler would not always terminate the actual execution properly. It is actually rather bizarre that there is no proper way of doing so.

A bit more research will certainly be done on that problem, to see if closing the communicator can be effective. But there is not that much advantage compare to a typical client-server application, and next development tests will be done on that.

Using the MPI profiling interface

How does the MPI profiling interface works? The answer is almost to easy. Finding how to use it is more complex.

The basic idea of the MPI profiling interface is simple: every single MPI function provides actually two entry points. One has the classical MPI_ prefix, the other has PMPI_. Thus, the whole idea is to overload the MPI_ ones, and call the corresponding PMPI_ in the middle. This approach gives full access to both parameters and return code.
Moreover the PMPI_ calls are part of the MPI standard definition (as far as I know...) and therefore are common to every implementations.

In order to test the MPI profiling interface I wrote down the simplest MPI code possible. Two files were needed, one for the wrapper, on for the program.

mpi_wrap.h

#ifndef MPI_WRAP
#define MPI_WRAP

int MPI_Init(int *argc, char ***argv);

#endif

mpi_wrap.c

#include "mpi_wrap.h"

#include <mpi.h>
#include <stdio.h>

int MPI_Init(int* argc, char ***argv)
{
  int ret;
  fprintf(stderr, "Prof: MPI_Init(...)");

  ret = PMPI_Init(argc, argv);

  return ret;
}

mpi_hello.c

#include <mpi.h>

#include "mpi_wrap.h"

int main()
{
  int rank=0, pop=0;

  MPI_Init(NULL, NULL);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &pop);

  if ( rank == 0 )
    printf("%d: I'm the master of %d puppets.\n", rank, pop);

  MPI_Finalize();

  return 0;
}

On Ness

Ness is the EPCC cluster used by MSc students, on Scientific Linux.

MPI installed: mpich-2

MPI C Compiler: pgcc

The first problem came from the compilation of the library. For a yet unknown reason ld doesn't want to link.

mpicc -c -fPIC mpi_wrap.c -o mpi_wrap.o
mpicc -shared -soname=libmpi_wrap.so -o libmpi_wrap.so mpi_wrap.o

Compilation ended by

/usr/bin/ld: /opt/local/packages/mpich2/1.0.5p4-ch3_sock-pgi7.0-7/lib/libmpich.a(init.o): relocation R_X86_64_32 against `MPIR_Process' can not be used when making a shared object; recompile with -fPIC
/opt/local/packages/mpich2/1.0.5p4-ch3_sock-pgi7.0-7/lib/libmpich.a: could not read symbols: Bad value.

Thus the static library approach was taken.

mpicc -c -fpic mpi_wrap.c -o mpi_wrap.o
ar rcs libmpi_wrap.a mpi_wrap.o

That compiled well.

Then comes the program compilation, that is straightforward.

mpicc -c -I. mpi_hello.c -o mpi_hello.o
mpicc mpi_hello.o -L. -lmpi_wrap -o mpi_hello

And the result worked fine:

$> mpiexec -n 2 mpi_hello
mpiexec: running on ness front-end; timings will not be reliable.
Prof: MPI_Init(...)
Prof: MPI_Init(...)
0: I'm the master of 2 puppets.

At home

My home desktop machine is using a Gentoo/Linux installation.

MPI installed: OpenMPI

MPI C Compiler: gcc

Compiling the dynamic library worked:

mpicc -c -fpic mpi_wrap.c -o mpi_wrap.o
mpicc -shared -Wl,-soname,libmpi_wrap.so mpi_wrap.o -o libmpi_wrap.so

And compiling the executable too:

mpicc -c -I. mpi_hello.c -o mpi_hello.o
mpicc -L. -lmpi_wrap mpi_hello.o -o mpi_hello

Of course as the dynamic library approach is used, the LD_LIBRARY_PATH environment variable has to be set from the directory where the .so is:

export LD_LIBRARY_PATH=`pwd`

Finally running works as well:

$> mpiexec -n 2 mpi_hello
Prof: MPI_Init(...)
Prof: MPI_Init(...)
0: I'm the master of 2 puppets.

Discussion

It is rather strange that Ness doesn't want to link as a dynamic library. Further investigation will be done on that problem, in order to find an answer.

Using a statically linked library offers the advantage of simplicity: no need to set up the LD_LIBRARY_PATH but increases the size of the executable, especially when the tool will include the graphical interface.

Thus the advantages of the dynamically linked library are the reversed, saving executable size as the expense of few configuration.

As far as possible I will try to use the dynamically linked library approach, as the graphical interface will certainly contains a lot of code, that is not directly needed into the program. But the library has to be present on a common ground if used on a cluster, and this will be something I need to investigate further on.

References

No real references here, but just some websites that helped me remember how to create libraries. And of course how to use the MPI profiling interface.

Creating a shared and static library with the gnu compiler [gcc] - René Nyffenegger

Open MPI FAQ: Performance analysis tools