Configuring Theano so that it doesn't directly crash when a GPU memory allocation fails

时光怂恿深爱的人放手 提交于 2019-12-09 23:41:54

问题


When a Theano script tries to obtain more GPU memory than currently available, it immediately crashes:

Error allocating 26,214,400 bytes of device memory (out of memory). 
Driver report 19,365,888 bytes free and 1,073,414,144 bytes total 

Is there any way to configure Theano so that it doesn't directly crash when a GPU memory allocation fails? E.g., it could instead retry every X seconds, and give up after Y tries.

(One use case: I have several Theano scripts running on the same GPU and using 90% of its memory only 5% of the time)

来源:https://stackoverflow.com/questions/38724733/configuring-theano-so-that-it-doesnt-directly-crash-when-a-gpu-memory-allocatio

标签
易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!