重置GPU显存 Reset GPU memory after CUDA errors

nvidia-smi --gpu-reset

Although it should be unecessary to do this in anything other than exceptional circumstances, the recommended way to do this on linux hosts is to unload the nvidia driver by doing

$ rmmod nvidia

 

1

$ rmmod nvidia 

with suitable root privileges and then reloading it with

$ modprobe nvidia

 

1

$ modprobe nvidia

If the machine is running X11, you will need to stop this manually beforehand, and restart it afterwards. The driver intialisation processes should eliminate any prior state on the device.

This answer has been assembled from comments and posted as a community wiki to get this question off the unanswered list for the CUDA tag

check what is using your GPU memory with

sudo fuser -v /dev/nvidia*

 

1

sudo fuser -v /dev/nvidia*

Your output will look something like this:

USER PID ACCESS COMMAND /dev/nvidia0: root 1256 F...m Xorg username 2057 F...m compiz username 2759 F...m chrome username 2777 F...m chrome username 20450 F...m python username 20699 F...m python

 

1

                     USER        PID  ACCESS COMMAND

2

/dev/nvidia0:        root       1256  F...m  Xorg

3

                     username   2057  F...m  compiz

4

                     username   2759  F...m  chrome

5

                     username   2777  F...m  chrome

6

                     username   20450 F...m  python

7

                     username   20699 F...m  python

Then kill the PID that you no longer need on htop or with

sudo kill -9 PID.

 

1

sudo kill -9 PID.

In the example above, Pycharm was eating a lot of memory so I killed 20450 and 20699.

来源： https://stackoverflow.com/questions/15197286/how-can-i-flush-gpu-memory-using-cuda-physical-reset-is-unavailable/46597252

来自为知笔记(Wiz)

来源：https://www.cnblogs.com/jins-note/p/11948927.html