Debug iPython Parallel engines in cluster

折月煮酒 提交于 2019-12-12 01:07:18

问题


Related to this thread...

I am trying to track down a bug in which the results from processing on an iPython cluster do not match what happens when the same process is run locally. Even when the iPython cluster is entirely local, and the CPU is simply running multiple engines.

I cannot seem to figure out how to log data as it is being processed on the engines. Print statements don't work and even when I try to have each engine write to a separate file, the file is created but nothing is written to it.

There must be a way to debug code running on the iPython parallel engines.


回答1:


Not sure why, but I narrowed the problem and workaround down to the fact that I am using cython and compiling the .pyx files before running the program.

For some reason the cython cdef init of my float variables was not being done properly on the engines however it was being done correctly when I ran outside of the Client() queue.

Changing these variables to be normal python variables solved the problem. Though it does not seem like this should have to happen. Anyone that can shed more light on this?



来源:https://stackoverflow.com/questions/21268990/debug-ipython-parallel-engines-in-cluster

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!