Celery dies with DBPageNotFoundError

问题:

I have 3 machines with celery workers and rabbitmq as a broker, one worker is running with beat flag, all of this is managed by supervisor, and sometimes celery dies with such error. This error appears only on beat worker, but when it appears, workers on all machines dies. (celery==3.1.12, kombu==3.0.20)

[2014-07-05 08:37:04,297: INFO/MainProcess] Connected to amqp://user:**@192.168.15.106:5672// [2014-07-05 08:37:04,311: ERROR/Beat] Process Beat Traceback (most recent call last): File "/var/projects/env/local/lib/python2.7/site-packages/billiard/process.py", line 292, in _bootstrap self.run() File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 527, in run self.service.start(embedded_process=True) File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 453, in start humanize_seconds(self.scheduler.max_interval)) File "/var/projects/env/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 322, in __get__ value = obj.__dict__[self.__name__] = self.__get(obj) File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 491, in scheduler return self.get_scheduler() File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 486, in get_scheduler lazy=lazy) File "/var/projects/env/local/lib/python2.7/site-packages/celery/utils/imports.py", line 53, in instantiate return symbol_by_name(name)(*args, **kwargs) File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 357, in __init__ Scheduler.__init__(self, *args, **kwargs) File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 184, in __init__ self.setup_schedule() File "/var/projects/env/local/lib/python2.7/site-packages/celery/beat.py", line 376, in setup_schedule self._store['entries'] File "/usr/lib/python2.7/shelve.py", line 121, in __getitem__ f = StringIO(self.dict[key]) File "/usr/lib/python2.7/bsddb/__init__.py", line 270, in __getitem__ return _DeadlockWrap(lambda: self.db[key])  # self.db[key] File "/usr/lib/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap return function(*_args, **_kwargs) File "/usr/lib/python2.7/bsddb/__init__.py", line 270, in <lambda> return _DeadlockWrap(lambda: self.db[key])  # self.db[key] DBPageNotFoundError: (-30985, 'DB_PAGE_NOTFOUND: Requested page not found') 

回答1:

I've ran into this issue and the cause was a corrupted db file (usually named "celerybeat-schedule"). Solution would be to delete the existing db file and restart the process.

Relavent:bsddb.db.DBPageNotFoundError https://mail.python.org/pipermail/python-list/2009-October/554552.html

回答2:

I had to remove some temp files in the /tmp directory. One was named celeryd-<NAME_OF_WORKER>-state and also celeryd-<NAME_OF_WORKER>-state-renamed. After removing those and I was able to restart my affected worker.

文章来源: Celery dies with DBPageNotFoundError