Python multiprocessing.pool sequential run of processes

问题

I am pretty new to python and the 'multiprocessing' module in particular. However, I have managed to write a very simple script to run multiple processes (say 100) on 24 cpus. However, I have noticed that the process are not run sequentially but instead randomly. Is there a way for the processes to be run sequentially. Here is my code:

#!/usr/bin/env python

import multiprocessing
import subprocess




def prcss(cmd):
  sbc = subprocess.call
  com = sbc(cmd, shell='True')
  return (com)


if __name__=='__main__':

  cmd = []
  for j in range(1,11):
    for i in range(10):
      sis = '~/codes-paul/sisyphus/sisyphus '+str(j)+'/sisyphus.setup > '+str(j)+'/out'+str(i)+'.dat'
      cmd.append(sis)


  pool=multiprocessing.Pool(processes=24)
  pool.map(prcss,cmd)

After I have run the python code, I do 'ps -ef | grep myname'. Instead of getting:

'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat'
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out1.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out2.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat
.
.
.
.
.I am getting:
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out0.dat.dat'
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out3.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out6.dat.dat
'/bin/sh -c ~/codes-paul/sisyphus/sisyphus > 1/sisyphus.setup > 1/out9.dat.dat 
.
.
.
.

Any idea why the commands are not run sequentially?

回答1:

Since you are creating a pool of processes, the commands are actually started sequentially, but you have no guarantee on which process is going to finish first. You will notice that the order will vary almost every time you run your code.

来源：https://stackoverflow.com/questions/27282538/python-multiprocessing-pool-sequential-run-of-processes

标签

python-multiprocessing