How to Consume an mpi4py application from a serial python script

Question

I tried to make a library based on mpi4py, but I want to use it in serial python code.

$ python serial_source.py

but inside serial_

Answer 1

This is indeed possible and is in the documentation of mpi4py in the section Dynamic Process Management. What you need is the so called Spawn functionality which is not available with MSMPI (in case you are working with Windows) see also Spawn not implemented in MSMPI.

Example

The first file provides a kind of wrapper to your function to hide all the MPI stuff, which I guess is your intention. Internally it calls the "actual" script containing your parallel code in 4 newly spawned processes.

Finally, you can open a python terminal and call:

from my_prog import parallel_fun

parallel_fun()
# Hi from 0/4
# Hi from 3/4
# Hi from 1/4
# Hi from 2/4
# We got the magic number 6

my_prog.py

import sys
import numpy as np
from mpi4py import MPI

    def parallel_fun():
        comm = MPI.COMM_SELF.Spawn(
            sys.executable,
            args = ['child.py'],
            maxprocs=4)

        N = np.array(0, dtype='i')

        comm.Reduce(None, [N, MPI.INT], op=MPI.SUM, root=MPI.ROOT)

        print(f'We got the magic number {N}')

Here the child file with the parallel code:

child.py

from mpi4py import MPI
import numpy as np


comm = MPI.Comm.Get_parent()

print(f'Hi from {comm.Get_rank()}/{comm.Get_size()}')
N = np.array(comm.Get_rank(), dtype='i')

comm.Reduce([N, MPI.INT], None, op=MPI.SUM, root=0)