Can't start Celery worker on Windows 10 with “PicklingError”

问题:

I have a simple test code that runs successfully on Linux, but it won't run on my windows 10 x64 computer.

When I tried to start a celery worker, it complained about the unrecoverable error: PicklingError. (Celery version: 3.1.20)

In my celery config, I've set the serialization to 'json', but it still didn't help at all.

CELERY_RESULT_SERIALIZER = 'json' CELERY_TASK_SERIALIZER = 'json' CELERY_ACCEPT_CONTENT = ['json'] 

Here is the full error message:

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)  Traceback (most recent call last):   File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start     self.blueprint.start(self)      File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start     step.start(parent)   File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start     return self.obj.start()   File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start     self.on_start()   File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start     **self.options)   File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__     self._create_worker_process(i)   File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process     w.start()   File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start     self._popen = Popen(self)   File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__     dump(process_obj, to_child, HIGHEST_PROTOCOL)   File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump     ForkingPickler(file, protocol).dump(obj)   File "C:\Python27\lib\pickle.py", line 224, in dump     self.save(obj)   File "C:\Python27\lib\pickle.py", line 331, in save     self.save_reduce(obj=obj, *rv)   File "C:\Python27\lib\pickle.py", line 401, in save_reduce     save(args)   File "C:\Python27\lib\pickle.py", line 286, in save     f(self, obj) # Call unbound method with explicit self   File "C:\Python27\lib\pickle.py", line 562, in save_tuple     save(element)   File "C:\Python27\lib\pickle.py", line 286, in save     f(self, obj) # Call unbound method with explicit self   File "C:\Python27\lib\pickle.py", line 548, in save_tuple     save(element)   File "C:\Python27\lib\pickle.py", line 331, in save     self.save_reduce(obj=obj, *rv)   File "C:\Python27\lib\pickle.py", line 401, in save_reduce     save(args)   File "C:\Python27\lib\pickle.py", line 286, in save     f(self, obj) # Call unbound method with explicit self   File "C:\Python27\lib\pickle.py", line 548, in save_tuple     save(element)   File "C:\Python27\lib\pickle.py", line 286, in save     f(self, obj) # Call unbound method with explicit self   File "C:\Python27\lib\pickle.py", line 649, in save_dict     self._batch_setitems(obj.iteritems())   File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems     save(v)   File "C:\Python27\lib\pickle.py", line 331, in save     self.save_reduce(obj=obj, *rv)   File "C:\Python27\lib\pickle.py", line 396, in save_reduce     save(cls)   File "C:\Python27\lib\pickle.py", line 286, in save     f(self, obj) # Call unbound method with explicit self   File "C:\Python27\lib\pickle.py", line 748, in save_global     (obj, module, name)) PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module 

回答1:

I ran into the same problem. What was weird is that the problem only existed on Windows, on Linux Celery was running without any problems. Turns out I needed to pass the configuration module as name not as object:

app.config_from_object('celeryconfig')

instead of

app.config_from_object(celeryconfig)

Explanation from the Celery docs:

Tip

Using the name of a module is recommended as this means that the module doesn’t need to be serialized when the prefork pool is used. If you’re experiencing configuration pickle errors then please try using the name of a module instead.

Apparently Celery needs to pickle the configuration when it's passed as an object which fails on Windows. If passed as module name it works. Thanks for pointing me in the right direction @JoyLy!

文章来源: Can't start Celery worker on Windows 10 with “PicklingError”