Meeting 6 [25/05] and how to use the tool

The meeting

Even thought a month came by, few new functionalities appeared into the project. Firstly because it was revision and examination period. During that period effort was made to insure the continuation of the project as design and researches. Some ideas on how to implement the synchronisation and possible functionalities were discuss (like the communicator registration).

The next step is to generate documentation for the project, as a proper webpage (and because some part of the code isn't much commented - as it was evolving quickly). Effort has to be make to make the project's folder organised. And some implementation will be carried on:

• communicator registration, to organise the MPI_Request saving
• syncrhonous communication (wait for the user to click continue before actually performing the MPI function

Was already in place the basis to do both implementation (MPI_Request are saved into a linked list and the backbone of the synchronisation is implemented - but not functional).

Using the tool

This tool aims to help people learning MPI behaviour. The sources have therefore been open to the "public". The first attempt was on an internal machine - Ness - that didn't work correctly. Therefore the project will be registered on source-forge, as it was planned, in advance.


This part is to explain how to use the current version of the code, and shouldn't change much in the future releases.

The project is composed of a library - the profiler - and an executable - the interface. The project should be organised into folders, one per deliverable. And should include tests. A general Makefile should be available to compile each of the deliverable, and a configure script may be available to automatise the variable generation (installation path, MPI flags to compile from the MPI compilers, Qt path, ...).

Compiling the profiler

Compiling the profiler requires:

• a C MPI implementation
• a C compiler

The profiler is available in both static or dynamic linking format, as only the linking stage changes. It is important for the user to be able to choose one or the other, as it appeared some MPI installation do not accept another type of library to use the MPI profiling interface.


Either mode could be compiled and installed, but note that if both are installed, it appears that dynamic linking is used by default.


Running make static or make dynamic should compile and install the library, by default in a local install folder composed of the classical lib and includes folders.

Compiling the interface

Compiling the interface requires:

• Qt 4.6 or later (note that Qt 4.7 was used but none of the used functionalities where introduced on that release).
• an C++ MPI implementation that supports multi-threading (see a previous note).
• a C++ compiler
• the headers from the profiler

The interface should be compilable from the main Makefile. A typical Qt project needs a project file to be generated that will generate the Makefile to compile it. Normally this process should be automatic, as the main Makefile should do so. If a configure script is available it should handles the variable generation, otherwise some variable needs to be set up:

• INSTALL_ROOT should contains the path to the installation folder (default: ../install as it is relative to the interface folder where it is built).
• MPI_INCLUDE should contain the path to the MPI headers. It can be retrieved by using mpicc -showme and is generally like -I/usr/local/include. However the -I should be REMOVED from the project option as QMake will generate it automatically.
• MPI_LINK should contain the linking options given by mpicc -showme and is generally like -pthread -L/usr/local/lib -lmpi_cxx -lmpi -lopen-rte -lopen-pal -ldl -Wl,--export-dynamic -lnsl -lutil -lm -ldl.
• MPI_EXTRA_FLAGS should be set up to -DMPICH_IGNORE_CXX_SEEK when using MPICH2 to avoid conflict with standard C++ file handling.

When the project file is done, and named as mpidisplay.pro, running make display should take care of the 2 compilation steps and of the installation. Nonetheless the steps are:

• The generation of the Makefile qmake mpidisplay.pro. You can specify the previously stated variables in the command line or in the file itself (example: qmake mpidisplay.pro INSTAL_ROOT=../install).
• Compiling the executable with the generated Makefile: make -f Makefile.qt
• Installing the executable is done by calling make -f Makefile.qt install

Using a MPI program with the library

Compiling

In order to compile the library with the profiler options, you need to know where the profiler library is installed. Let's assume ~/local/, meaning that the library is in ~/local/lib and the headers in ~/local/includes. The location of the mpidisplay interface isn't important yet, but it certainly in ~/local/bin.


Note that to compile - even as a dynamic library - you do not need the LD_LIBRARY_PATH to be updated, but you will need it to run the software later. You don't need to update the variable if you use static linking as the library is completely added to your executable. To set the path simple execute export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/local/lib in your shell - or add it to your bashrc file.


Compiling your program is done exactly the same way than any other MPI program with additional library. You need to add the headers path to the compiler flags and the library location and name to the linking flags. In a generic Makefile this is done by adding CFLAGS+=-I~/local/includes and LDFLAGS+=-L~/local/lib -lmpi_wrap.


The only source modification is then to add to your code files:

#include <mpi_wrap.h>

In theory you can even remove the flags if you don't want to use the library, but be aware that the compiler might complain about not finding "mpi_wrap.h". Therefore you can define a precompiler macro WITH_MPIWRAP by adding CFLAGS+=-I~/local/includes -DWITH_MPIWRAP and doing your include as

#ifdef WITH_MPIWRAP
#include <mpi_wrap.h>
#endif

Running

As stated before your LD_LIBRARY_PATH should be updated if you are using a dynamic linking. Otherwise simply run your MPI program as usual. Assuming your program executable is called ring you usually ran mpiexec -n 4 ring to run with 4 MPI processes. With the library it is the same!

The profiler library will write the port on the standard output by default. But you can add command line arguments to define another way:

• Standard output with --port-in-stdout
• Standard error output with --port-in-stderr
• A text file with --port-in-file file

Start the interface

Starting message box of the interface (GNU/Linux Gnome 3)

In order to start the mpidisplay interface you need to add its location to the PATH with the same technique than the LD_LIBRARY_PATH: export PATH=$PATH:~/local/bin. Then simply run mpidisplay to see the connection window. If you exported the port on the standard output move to the "manual" tab and write the ports in the fields. You can change the number of processors in the list - and the order of the port does not matter. If you used a text file, click the button and select it; the ports information will be loaded on the text edit area underneath if you need edition (and the number of processors should be updated).

Then simply click OK to start the interface.


You can find for the moment 2 main information in the interface: the number of calls to a sample of MPI routines and the time spent in them.

Profiler-Interface communication

Remember

Just to remember to the reader: the actual code is working with a client-server communication. The server part - the MPI profiler - sends information to the client - the interface - through a MPI interconnect done with MPI_Open_port.


In order to do so a protocol has to be defined for the messages.

Client-Server organisation

Connection

The Profiler-Interface organisation (4 MPI processes)

The connection of the profiler and the interface is done via the MPI_Open_port function, that opens and gives a port address. Each MPI process from the profiler publishes its own port, and therefore the interface has to connect to every single one of them. That actually means there is n servers and 1 client. It is unusual of a client-server model, as usually only 1 server delivers information to several clients. Nonetheless the profiler processes are the servers, as they are the ones that publish an accessible address.

With OpenMPI

OpenMPI provides an address that looks like:


117112832.0;tcp://192.168.1.71:36441+117112833.0;tcp://192.168.1.71:42986:300


An early attempt to guess the port was unsuccessful. Some part may change from a process to another without other logic than available resource (the port for example).

But a problem came when connecting in the interface. As the previous diagram shows it, several profilers connect to a single interface process. What is not shown is that the interface has in fact 1 thread per profiler, to deal with the communication. The data is then centralised in a single GUI. This approach is a typical Hybrid MPI programming approach. Therefore the interface has to initialise the MPI environment with MPI_Init_thread (rather than MPI_Init) and ask for a MPI_THREAD_MULTIPLE initialisation. By default OpenMPI doesn't provide such support.


The solution is rather simple: recompile OpenMPI with the threading support:

./configure --enable-mpi-threads

With MPICH-2

The Ness machine provided a MPICH-2 implementation already installed. For some reasons it didn't support dynamic linking, but static one is fine. For this implementation the port string looks like:

tag=0 port=52970 description=ness.epcc.ed.ac.uk ifname=129.215.175.1

That is radically different of the OpenMPI one, showing once more that guessing the port isn't a interesting idea.

As MPICH-2 is natively installed on Ness with the multi-threading, the configuration option isn't yet known.

Retrieving the port

The profiler opens and publish a port. As a matter of fact, the user has to read the port and give them in input to the interface. In order to give as much freedom as possible to the user several ways of doing it are available:

• Printing the port to the standard output stream
• Printing the port to the standard error output stream
• Writing the ports into a defined file

This is achieved by giving information when calling MPI_Init on the MPI code. This could be achieved simply by providing command line arguments when calling mpiexec. The available arguments can be retrieved with (ring is the executable name):

$> ./ring --help
Profiler of an MPI program\nUse a MPI visualisation GUI to see information

Possible options:
--port-in-stdout [default]
   write the port into the standard output
--port-in-stderr
   write the port into the standard error output
--port-in-file file
   write the port into the file using MPI-I/O

Note that only the last given option is used

--help
   display that help

To use the file writing functionality simply start you program like:

$> mpiexec -n 4 ring --port-in-file port.txt

Note: so far adding the option manually as 2D array of char doesn't work, and no further looking as been made to make it work.

Writing each process' port in a single file

In order to write each process' port in a single file the MPI I/O functions are used. The standard defines several ways of doing so. In that case a simple subarray is defined with the size of the port as a base length. MPI I/O writes data as a whole line into a file, as this stores characters a new line is created for each port. The interface can therefore read the file line by line to find every port and know the number of started processes.

Extract of child_comm.c

if ( port == INFILE )
    {
      MPI_Datatype subarray;
      MPI_File file_ptr;
      int smallarray, bigarray, stride;

      smallarray = (strlen(port_name)+1);
      bigarray = world_size*smallarray;
      stride = world_rank*smallarray;

      fprintf(stderr, "!profiler(%d)! will write his port in '%s'\n", world_rank, file);

      MPI_Type_create_subarray(1, &bigarray, &smallarray, &stride, MPI_ORDER_C, INTRA_MESSAGE_MPITYPE, &subarray);
      MPI_Type_commit(&subarray);

      if ( MPI_File_open(MPI_COMM_WORLD, file, MPI_MODE_WRONLY|MPI_MODE_CREATE, MPI_INFO_NULL, &file_ptr) != MPI_SUCCESS )
 {
   fprintf(stderr, "!profiler(%d)! failed to open file '%s'. ABORTING\n", world_rank, file);
   MPI_Abort(MPI_COMM_WORLD, -1);
 } 

      if ( MPI_File_set_view(file_ptr, 0, INTRA_MESSAGE_MPITYPE, subarray, "native", MPI_INFO_NULL) != MPI_SUCCESS )
 {
   fprintf(stderr, "!profiler(%d)! failed to set the file view! ABORTING\n", world_rank);
   MPI_Abort(MPI_COMM_WORLD, -1);
 }

      if ( MPI_File_write_all(file_ptr, strcat(port_name, "\n"), smallarray, INTRA_MESSAGE_MPITYPE, MPI_STATUS_IGNORE) != MPI_SUCCESS )
 {
   fprintf(stderr, "!profiler(%d)! failed to write '%s'. ABORTING\n", world_rank, file);
   MPI_Abort(MPI_COMM_WORLD, -1);
 }

      MPI_File_close(&file_ptr);
    }

Communication

The profiler side

As far as the profiler is concerned, the communication with the interface could be either synchronous or asynchronous. The current implementation uses MPI_Ssend as simple choice, but later version could use asynchronous call and waiting before the next one is done. Or even deal with a request list to wait for.

The profiler uses internal functions defined into child_comm.h to communicate with the interface.

child_comm.h

#ifndef CHILDCOMM
#define CHILDCOMM

#include "intra_comm.h"

extern int world_rank;
extern double global_time;

typedef enum PortType { STDOUT, STDERR, INFILE } PortType;

int start_child(int world_size, PortType port_type, char* file);
int alive_child();
int sendto_child(Intra_message* message);
int wait_child(double time_in);

#endif // CHILDCOMM

intra_comm.h

#ifndef INTRA_COMM
#define INTRA_COMM

#define INTRA_MESSAGE_SIZE 64
typedef char Intra_message;

#define INTERCOMM_TAG 0

#define PROFNAME "!profiler!"

#ifdef __cplusplus
#define INTRA_MESSAGE_MPITYPE MPI::CHAR
#else
#define INTRA_MESSAGE_MPITYPE MPI_CHAR
#endif

/*
 * ACTIONS
 */

typedef enum Message { MESSAGE_INIT,
               MESSAGE_Ssend,
               MESSAGE_Bsend,
               MESSAGE_Issend,
               MESSAGE_Recv,
               MESSAGE_Irecv,
               MESSAGE_Wait,
               MESSAGE_QUIT } Message;

#endif // INTRA_COMM

The functions' name are explicit, and the intra_comm.h header defines the actual protocol information: it is therefore used by both profiler and interface. The actual sending is done by character stings, renamed as Intra_message. As the interface is coded in C++ the INTRA_MESSAGE_MPITYPE is defined using both C and C++ MPI standard definitions.


The message is composed of several fields, all separated by a space, which always includes main fields:

• action::enum Message the occurring action
• time in::double the Unix time when entering the MPI function
• time out::double the Unix time when returning the MPI function

But each Message has its own information to add as well, after the main ones. For example a MPI_Ssend also encapsulate:

• communicator::unsigned int the communicator unique number - not implemented yet
• destination::int the destination process

And some more information could be added as needed. Each MPI function defines its own optional fields in his own call to sendto_child().

The information are written using standard C I/O calls:

sprintf(message, "%d %lf %lf %d\0", MESSAGE_Ssend, time_in, time_out, dest);

The interface side

Starting message box of the interface (GNU/Linux Gnome 3)

On the interface side the profilers' port could be defined either manually or by reading the file written as explained before. When this is done, one thread per process is started and their duty is to communicate with the profiler (the object is therefore called MPIWatch). The MPIWatch object is only responsible for receiving (and sending) information to the profiler, therefore each of them is attached to a Monitor object, that is responsible of the analyse of messages. In order to communicate the MPIWatch pushes arriving message onto a stack and signal to the Monitor that new messages are available. The Monitor then analyse the message and display information in the according places.

The couple MPIWatch - Monitor was done for logical purposes:

1. Only the MPIWatch is actually aware of the MPI functions needed to sends and receive information to the profiler. If in the future another system is used, only this class has to be changed.
2. Only the MPIWatch needs a separated thread, dealing with the messages contents is done on the main thread.
3. Only the Monitor has the knowledge of what a message contains. New protocol functionnalities does not affect the way to transfer data between profiler and interface
4. Only the Monitor knows about the GUI, that are shared "windows" among the several monitors.

As the interface is implemented in C++, the standard stream library is used to decapsulate the messages. The main fields are extract for each messages, and the according to the message action each additional information.

Extract of monitor.cpp

QString m = watcher->pop_pool();
std::istringstream stream(m.toStdString());
int message;
double time_in, time_out;

stream >> message >> time_in >> time_out;

switch(message)
{
        /* ... */

    case MESSAGE_Bsend:
        // adds to call counts
        statWidget->addTo(proc, N_Bsend); 
        // add time info
        statWidget->addTo(proc, T_Bsend, time_out-time_in); 
        break;

        /* ... */
}

Conclusion

The Profiler-Interface communication is done on two levels. The first one is the actual communication, done through MPI. This requires a port opening and publish mechanism, that the user has to give as an input to the profiler.

But the communication is also what information is sent. This is generated by each overloaded MPI function, and is analysed in the interface side by a Monitor object.

Decoupling the communication on these two levels allows an abstraction of actually sending and analysing the information.