How can I solve 'ran out of gpu memory' in TensorFlow

Question

I ran the MNIST demo in TensorFlow with 2 conv layers and a full-conect layer, I got an message that \'ran out of memeory trying to allocate 2.59GiB\' , but it shows that to

Answer 1

From TensorFlow guide

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
    try:
        tf.config.experimental.set_virtual_device_configuration(gpus[0],
       [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Virtual devices must be set before GPUs have been initialized
        print(e)

Adjust memory_limit=*value* to something reasonable for your GPU. e.g. with 1070ti accessed from Nvidia docker container and remote screen sessions this was memory_limit=7168 for no further errors. Just need to make sure sessions on GPU cleared occasionally (e.g. Jupyter Kernel restarts).