RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

Question

On ubuntu14.04,I use pytorch with cudnn.This problem happened:

Traceback (most recent call last):
  File "main.py", line 58, in <module>
    test_detect(test_loader, nod_net, get_pbb, bbox_result_path,config1,n_gpu=config_submit['n_gpu'])
  File "/home/ubuntu/nndl/DSB2017/test_detect.py", line 52, in test_detect
    output = net(input,inputcoord)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 58, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/nndl/DSB2017/net_detector.py", line 102, in forward
    out = self.preBlock(x)#16
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 351, in forward
    self.padding, self.dilation, self.groups)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 119, in conv3d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

I have google it for severial hours an am really confused.What made this happen?

Answer 1

I just encountered this problem on ubuntu16.04 and solved it. My solution was to run

sudo rm -rf ~/.nv

and then reboot.