Too many open processes when using multiprocessing. How close earlier processes?

问题

I'm new to multiprocessing and need it for processing OpenStreet data. I use Python 3.5. I'm aware that there are some questions touching the topic. But I would prefer a solution with the Process class. Of course, other suggestions are welcome.

import overpy
api = overpy.Overpass()
result = api.query("""area[name="Aachen"]->.a;(rel(area.a)["building"];way(area.a)["building"];);(._;>;);out;""")

import multiprocessing as mp

def transfer(i,send_end):
    way = result.ways[i]

    bldg =  {'id':way.id,
             'nodeid':[way.nodes[h].id for h in range(len(way.nodes))],
             'coords':[(way.nodes[j].lat, way.nodes[j].lon) for j in range(len(way.nodes))],
            'tags':way.tags} 
    send_end.send(bldg)


if __name__=='__main__':
    jobs = []
    pipe_list = []
    for i in range(10000):
        recv_end, send_end = mp.Pipe(False)
        p = mp.Process(target=transfer,args=(i,send_end))
        jobs.append(p)
        pipe_list.append(recv_end)
        p.start()
   for proc in jobs:
       proc.join()

   result_list = [x.recv() for x in pipe_list]

The problem is that there are around 60.000 objects, and thus so many processes will be started. This overwhelms my system and triggers:

OSError: [Errno 24] Too many open files

Has anybody an idea how to liberate resources before the end of the loop? I tried to use join() within the loop but then, I don't know why, it won't use all the cores.

来源：https://stackoverflow.com/questions/47426410/too-many-open-processes-when-using-multiprocessing-how-close-earlier-processes