问题
I am trying to run a neural network with pycaffe on gpu.
This works when I call the script for the first time. When I run the same script for the second time, CUDA throws the error in the title.
Batch size is 1, image size at this moment is 243x322, the gpu has 8gb RAM.
I guess I am missing a command that resets the memory?
Thank you very much!
EDIT:
Maybe I should clarify a few things: I am running caffe on windows.
When i call the script with python script.py, the process terminates and the gpu memory gets freed, so this works.
With ipython, which I use for debugging, the GPU memory indeed does not get freed (after one pass, 6 of the 8 bg are in use, thanks for the nvidia-smi suggestion!)
So, what I am looking for is a command I can call from within pyhton, along the lines of:
run network
process image output
free gpu memory
回答1:
Your GPU memory is not getting freed. This happens when the previous process is stopped but not terminated. See my answer here.
回答2:
This happens when you run out of memory in the GPU. Are you sure you stopped the first script properly? Check the running processes on your system (ps -A in ubuntu) and see if the python script is still running. Kill it if it is. You should also check the memory being used in your GPU (nvidia-smi).
来源：https://stackoverflow.com/questions/40090892/check-failed-error-cudasuccess-2-vs-0-out-of-memory