Error in Python multiprocessing process

问题

I am trying a write a python code having multiple processes whose structure and flow is something like this:

import multiprocessing
import ctypes
import time
import errno
m=multiprocessing.Manager()
mylist=m.list()
var1=m.Value('i',0)
var2=m.Value('i',1)
var3=m.Value('i',2)
var4=m.Value(ctypes.c_char_p,"a")
var5=m.Value(ctypes.c_char_p,"b")
var6=3
var7=4
var8=5
var9=6
var10=7
def func(var1,var2,var4,var5,mylist):
    i=0
    try:
        if var1.value==0:
            print var2.value,var4.value,var5.value
            mylist.append(time.time())
        elif var1.value==1:
            i=i+2
            print var2.value+2,var4.value,var5.value
            mylist.append(time.time())
    except IOError as e:
        if e.errno==errno.EPIPE:
            var3.value=var3.value+1
            print "Error"
def work():
    for i in range(var3.value):
        print i,var6,var7,va8,var9,var10
        p=multiprocessing.Process(target=func,args=(var1,var2,var4,var5,mylist))
        p.start()
work()

When I run this code, sometimes it works perfectly, sometimes it does not run for exact amount of loop counts and sometimes I get following error:

0
1
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "dummy.py", line 19, in func
    if var1.value==0:
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 1005, in get
    return self._callmethod('get')
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 722, in _callmethod
    self._connect()
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 709, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 149, in Client
    answer_challenge(c, authkey)
  File "/usr/lib64/python2.6/multiprocessing/connection.py", line 383, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
EOFError

What does this error mean? What wrong am I doing here? What this error indicates? Kindly guide me to the correct path. I am using CentOS 6.5

回答1:

Working with shared variables in multiprocessing is tricky. Because of the python Global Interpreter Lock (GIL), multiprocessing is not directly possible in Python. When you use the multiprocessing module, you can launch several task on different process, BUT you can't share the memory. In you case, you need this so you try to use shared memory. But what happens here is that you have several processes trying to read the same memory at the same time. To avoid memory corruption, a process lock the memory address it is currently reading, forbidding other processes to access it until it finishes reading. Here you have 3 processes trying to evaluate var1.value in the first if loop of your func : the first process read the value, and the other are blocked, raising an error. To avoid this mechanism, you should always manage the Lock of your shared variables yourself. You can try with syntax:

var1=multiprocessing.Value('i',0) # create shared variable
var1.acquire() # get the lock : it will wait until lock is available
var1.value # read the value
var1.release() # release the lock

External documentation :

Locks : https://docs.python.org/2/librar/multiprocessing.html#synchronization-between-processes GIL : https://docs.python.org/2/glossary.html#term-global-interpreter-lock

来源：https://stackoverflow.com/questions/30348651/error-in-python-multiprocessing-process

标签

python

CentOS

python-multiprocessing