NotImplementedError: X is not picklable

问题

Using Python 3.x, I am trying to iterate over a dictionary of datasets (NetCDF4 datasets). They are just files...

I want to examine each dataset on a separate process:

def DoProcessWork(datasetId, dataset):
    parameter = dataset.variables["so2"]
    print(parameter[0,0,0,0])

if __name__ == '__main__':
    mp.set_start_method('spawn')
    processes = []
    for key, dataset in datasets.items():
        p = mp.Process(target=DoProcessWork, args=(key, dataset,))
        p.start()
        processes.append(p)

When I run my program, I get some message about 'pickable'

File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files (x86)\Python36-32\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "netCDF4\_netCDF4.pyx", line 1992, in netCDF4._netCDF4.Dataset.__reduce__ (netCDF4\_netCDF4.c:16805)
NotImplementedError: Dataset is not picklable

What am I doing wrong? How can I fix this? Could it be that opening the file is done on another process, and so I am getting an error because I am trying to pass data loaded on 1 process to another process?

回答1:

The multiprocessing needs to serialize (pickle) the inputs to pass them to the new proccess which will run the DoProcessWork. In your case the dataset object is a problem, see the list of what can be pickled.

A possible workaround for you would be using multiprocessing with another function which reads the dataset and calls DoProcessWork on it.

来源：https://stackoverflow.com/questions/43825150/notimplementederror-x-is-not-picklable

标签

python

python-3.x

pickle