Why does MPI_Iprobe return false when message has definitely been sent?

Question

I want to use MPI_Iprobe to test whether a message with a given tag is already pending.

However, the behaviour of MPI_Iprobe isn\'t quite as I was expecting. In the

Answer 1

You are using MPI_Isend in order to send the messages. MPI_Isend initiates an asynchronous (background) data transfer. The actual data transfer might not happen unless one of the MPI_Wait* or MPI_Test* calls has been made on the request. Some MPI implementations have (or can be configured so) background progression threads that will progress the send operation even if no wait/test has been done on the request, but one should not rely on such behaviour.

Simply replace MPI_Isend with MPI_Send or add MPI_Wait(&request); after the former (mind though that MPI_Isend + MPI_Wait immediately after is equivalent to MPI_Send).

MPI_Iprobe is intended to be used in busy waits, i.e.:

while (condition)
{
   MPI_Iprobe(...,&flag,...);
   if (flag)
   {
      MPI_Recv(...);
      ...
   }
   // Do something, e.g. background tasks
}

Real-life message transfers in actual MPI implementations are quite complicated things. Operations are usually split in multiple parts that are then queued. Executing that parts is called progression and it is done at various points in the MPI library, for example when a communication call is made or in background if the library implements a background progression thread. Calling MPI_Iprobe certainly does progress things but there is no guarantee that a single call will suffice. The MPI standard states:

The MPI implementation of MPI_PROBE and MPI_IPROBE needs to guarantee progress: if a call to MPI_PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI_PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI_IPROBE and a matching message has been issued, then the call to MPI_IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation.

Note the use of eventually. How progression is being done is very implementation-specific. Compare the following output from 5 consecutive calls to MPI_Iprobe (your original code + a tight loop):

Open MPI 1.6.5 w/o progression thread:

# Run 1
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1

# Run 2
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1

# Run 3
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0

No consistency between several executions of the same MPI program is observed and in the 3rd run the flag is still false after 5 invocations of MPI_Iprobe.

Intel MPI 4.1.2:

# Run 1
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1

# Run 2
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1

# Run 3
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 0
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1
After MPI_Iprobe, flag = 1

Obviously Intel MPI progresses things differently than Open MPI.

The difference between the two implementations can be explained by the fact that MPI_Iprobe is supposed to be a tiny probe and therefore it should take as less time as possible. Progression on the other hand takes time and in single-threaded MPI implementations the only point in time where progression is possible is the call to MPI_Iprobe (in that particular case). Therefore the MPI implementer has to decide how much is actually progressed by each call to MPI_Iprobe and strike a balance between the amount of work done by the call and the time it takes.

With MPI_Probe things are different. It is a blocking call and therefore it is able to constantly progresses until a matching message (and more specifically its envelope) appears.