mpi processes working in a burst

问题

I am using mpi4py to model a distributed application.

I have n processes accessing a shared file and writing some logs into the shared file during their execution. I notice that the logs are not uniformly written. Here is an example of how logs are written into the shared file:

process0.log0
process0.log1
process0.log2
process0.log3
process0.log4
process2.log0
process2.log1
process2.log2
process1.log0
process1.log1

Ideally it should be like:

process0.log0
process1.log0
process2.log0
process0.log1
process2.log1
process1.log1
process0.log2

Can anyone tell me what is possibly wrong with my implementation? I am writing into the file using Pickle module.

following is the function which dumps the log:

import pickle

log_file_name = "store.log"

def writeLog(data):
  try:
    with open(log_file_name,"a") as fp:
        pickle.dump(obj=data,file=fp)
  except:
    with open(log_file_name,"w") as fp:
        pickle.dump(obj=data,file=fp)

def readLog():
 data = []
 try:
    with open(log_file_name,"r") as fp:
        while True:
            data.append(pickle.load(fp))
    return data
 except EOFError:
    return data

All n processes access this function to dump the data

回答1:

There are lots of questions/answers out there that explain the phenomenon you're seeing here:

MPI - Printing in an order
Using MPI, a message appears to have been recieved before it has been sent
how do I print the log in order in MPI
Why does this MPI code execute out of order?)
Redirecting stdout from children spawned via MPI_Comm_spawn

Even though these are (mostly) talking about printing to the screen, the problem is the same. MPI is a distributed model which means that some processes will execute faster than others and it will probably be a different order every time depending on the workload/ordering of each process.

If ordering is important, you can use synchronization functions to enforce it or you can use something more fancy like MPI I/O for writing to files (not my specialty so I can't tell you much more about it).

来源：https://stackoverflow.com/questions/25378022/mpi-processes-working-in-a-burst

标签

python

mpi

pickle