Register a communicator

Why registering a communicator?

Registering a communicator was, originally, an idea developed to allow the user to filter communication occurring on a specific MPI communicator rather than on all of the used ones. But quickly during the tests it appears that the MPI profiling interface was used for every call of the library, meaning that when MPI_Ssend is redefined for example, even for a simple program (like the message in ring, 40 sends, 40 receives, 40 waits) the count of messages was enormous (certainly internal messages).

It became therefore important to filter the messages to register in the Interface, and in order to avoid network congestion, to do so on the Profiler side. The first way to reduce this number of messages is very simple: check for MPI_COMM_WORLD, that is the basic communicator used. But it doesn't provide the user any choice on monitoring a communicator or not.

int MPI_Ssend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
{
  int ret;

  ret = PMPI_Ssend(buf, count, datatype, dest, tag, comm);

  if ( comm == MPI_COMM_WORLD )
    {
      /* send information to the Interface */
    }

  return ret;
}

Registering Communicators

Registering from the user side

A mechanism has to be create to allow the user to register and unregister communicators at will. Few functionalities are visible from the user side:

• is a communicator registered?
• register a communicator
• unregister a communicator

Resulting in the following functions:

int Comm_is_registered(MPI_Comm comm);
int Comm_register(MPI_Comm comm, char* name);
int Comm_unregister(MPI_Comm comm);

When a user registers a communicator, a name is given, and this name is displayed in the Interface as the communicator name. By default MPI_COMM_WORLD is registered when the application starts, but further options may add the possibilities to avoid doing so.

Each communicator will have a unique identifier (an unsigned integer) that will be sent to the Interface. For each communicator sent to the interface this identifier will be given, allowing the GUI to sort information by communicator when needed.

Registering: use in the profiling interface

The actual registration is done via a linked list. When the user registers a communicator, the list is searched for it, and if it is not present, added. When a MPI call is processed, the list is searched as well, and if the communicator isn't present, no information is sent to the Interface. The MPI profiling function now looks like following.

int MPI_Ssend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)
{
  int ret;
  Register_Comm* commInfo = NULL;

  commInfo =  Comm_register_search(comm);

  ret = PMPI_Ssend(buf, count, datatype, dest, tag, comm);

  if ( commInfo )
    {
        /* send information to the Interface */
    }

  return ret;
}

Registering a communicator: internals

Communicator cells struct diagram

Registering a communicator looks relatively easy. The class diagram is relatively simple, though it is a double linked list (linking previous and next cells).

The MPI_Comm datatype could be considered as a simple int or long int as it represents the address of a communicator internally. In OpenMPI it is a structure. In MPICH2 it is defined as an int. Therefore a comparison like follows works.

int compare(MPI_Comm a, MPI_Comm b)
{
   return a == b;
}

Using MPI_Comm_compare

But comparing the datatype itself isn't very portable. The goal of that tool is to work with the MPI standard, not really using tricks from implementations to implementations. A function exists to compare two communicators: MPI_Comm_compare. According to the standard it returns:

• MPI_IDENT results if and only if comm1 and comm2 are handles for the same object (identical groups and same contexts).
• MPI_CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicators differ only by context.
• MPI_SIMILAR results of the group members of both communicators are the same but the rank order differs.
• MPI_UNEQUAL results otherwise.

while ( curr != NULL )
    {
      MPI_Comm_compare(curr->comm, comm, &res);

        switch(res)
        {
        case MPI_IDENT:
            fprintf(stderr, "got MPI_IDENT\n");
            break;
        case MPI_CONGRUENT:
            fprintf(stderr, "got MPI_CONGRUENT\n");
            break;
        case MPI_SIMILAR:
            fprintf(stderr, "got MPI_SIMILAR\n");
            break;
        case MPI_UNEQUAL:
            fprintf(stderr, "got MPI_UNEQUAL\n");
            break;
        }

   curr = curr->next;
}

Using Communicators attributes

In order to go further, the user should be able to register a communicator for some time, and then unregister it. But just maintaining a list of currently registered communicator, if an unregistered communicator is registered again, it will become a new communicator.



This behaviour could be avoided. Firstly a list "communicator registered in the past" could be created, and each time a communicator is created, a search is performed. This isn't a bad option as usually few communicators are used, but isn't very interesting in term of performance. The MPI standard defines attributes that can be attached to an object. In that case an attribute could be created when registering the communicator (storing its unique ID for instance). This attribute could be looked for just before the insertion in the list, and if it exists, retrieve the unique ID from it.

Conclusion and limitations

The simple mechanism (comparing as int) is used effectively to search through registered communicators, and be able to add/remove communicators. But a more portable way will be used in the future, in order to complain with the standard rather than adapting to implementors versions. The actual "retrieving" of information from a deleted communicator isn't implemented yet, and will certainly be useful for watching a particular moment of a code rather than the whole MPI program without creating a lot of communicators in the Interface.