Function call stack: keras_scratch_graph Error

Question

I am reimplementing a text2speech project. I am facing a Function call stack : keras_scratch_graph error in decoder part. The network architecture is from D

Answer 1

If you use Tensorflow-GPU, then add:

physical_devices = tf.config.experimental.list_physical_devices('GPU')
print("physical_devices-------------", len(physical_devices))
tf.config.experimental.set_memory_growth(physical_devices[0], True)

In addition, you can reduce your batch_size or change another computer or cloud services, like google colab, amazon cloud to run your codes because I think this is because the limitation of memory.

Answer 2

My situation is tensorflow sample code works fine in Google colab but not in my machine as I got keras_scratch_graph error.

Then i add this Python code at the beginning and it works fine.

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        # Restrict TensorFlow to only use the fourth GPU
        tf.config.experimental.set_visible_devices(gpus[0], 'GPU')

        # Currently, memory growth needs to be the same across GPUs
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Memory growth must be set before GPUs have been initialized
        print(e)

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process.

In some cases it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process.

For example, you want to train multiple small models with one GPU at the same time. By calling tf.config.experimental.set_memory_growth, which attempts to allocate only as much GPU memory in needed for the runtime allocations: it starts out allocating very little memory, and as the program gets run and more GPU memory is needed, we extend the GPU memory region allocated to the TensorFlow process.

Hope it helps!

Answer 3

it my case I had to update keras and tensorflow

pip install -U tensorflow keras 

Answer 4

You need to add this code at the top of your script, it works for me on TensorFlow 2.2.0

if tf.config.list_physical_devices('GPU'):
    physical_devices = tf.config.list_physical_devices('GPU')
    tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)
    tf.config.experimental.set_virtual_device_configuration(physical_devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4000)])

Answer 5

I think it's a thing about the gpu. look at the traceback:

File "/Users/ydc/dl-npm/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 572, in __call__
    return self._call_flat(args)

tf is calling on eager execution, which means that gpu will be used if the version is available. I had the same issue when I was testing a dense network:

inputs=Input(shape=(100,)
             )
x=Dense(32, activation='relu')(inputs)
x=Dense(32, activation='relu')(x)
x=Dense(32, activation='relu')(x)
outputs=Dense(10, activation='softmax')(x)
model=Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
t=tf.zeros([1,100])
model.predict(t, steps=1, batch_size=1)

... and it gave a similar traceback, also linking to eager execution. Then when I disabled gpu using the following line:

tf.config.experimental.set_visible_devices([], 'GPU')

... the code ran just fine. See if this would help solve the issue. Btw, does colab even support gpu? I didn't even know.

Answer 6

I was getting similar error. I reduced the batch size and the error disappeared. I don't know why but it worked for me. I am guessing something related to over stacking.