Celery tasks profiling

依然范特西╮ 提交于 2021-02-07 06:24:09

问题


As I can see in top utility celery procecess consume a lot of CPU time. So I want to profile it.

I can do it manually on developer machine like so:

python -m cProfile -o test-`date +%Y-%m-%d-%T`.prof ./manage.py celeryd -B

But to have accurate timings I need to profile it on production machine. On that machine (Fedora 14) celery is launched by init scripts. E.g.

service celeryd start

I have figured out these scripts eventually call manage.py celeryd_multi eventually. So my question is how can I tell celeryd_multi to start celery with profiling enabled? In my case this means add -m cProfile -o out.prof options to python.

Any help is much appreciated.


回答1:


I think you're confusing two separate issues. You could be processing too many individual tasks or an individual task could be inefficient.

You may know which of these is the problem, but it's not clear from your question which it is.

To track how many tasks are being processed I suggest you look at celerymon. If a particular task appears more often that you would expect then you can investigate where it is getting called from.

Profiling the whole of celery is probably not helpful as you'll get lots of code that you have no control over. As you say it also means you have a problem running it in production. I suggest you look at adding the profiling code directly into your task definition.

You can use cProfile.run('func()') as a layer of indirection between celery and your code so each run of the task is profiled. If you generate a unique filename and pass it as the second parameter to run you'll have a directory full of profile data that you can inspect on a task-by-task basis, or use pstats.add to combine multiple task runs together.

Finally, per-task profiling means you can also turn profiling on or off using a setting in your project code either globally or by task, rather than needing to modify the init scripts on your server.



来源:https://stackoverflow.com/questions/7473537/celery-tasks-profiling

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!