Buffered data was truncated after reaching the output size limit

Question

When I use Colaboratory to run my NIN model, it occurs an error in the output of training process which tells \"Buffered data was truncated after reaching the output size li

Answer 1

it's not relating to hardware RAM or GPU capacity.

Keras framework has a limitation for showing output info in console.

when you see this message, your process is going on in background but you can't see it.

If you use tensorflow as backend, write a Tensorboard callback in your Keras to see detailed output of your network.

https://keras.io/callbacks/#tensorboard

Answer 2

Even if RAM | GPU | DISK on colab is free, this error still comes because there is a limited memory for displaying output of a cell on colab. Assuming the memory limit is around 2Mb to 5Mb when we run many epochs(148+) during training, it tends to fill that memory and hence the output is truncated because there is no more memory left free to display the buffered epochs. However, the machine keeps running in the background and the output is processed but it is not displayed because of the buffered limit. You will still get your desired output.

One solution is not to use verbose=1 (use 0 instead).

Answer 3

I think this error is for out of memory. Your RAM or GPU memory was full and could not process new data. you can do two works: 1. Decrease your batch size. 2. Save your model in for example 60th epoch and close current program and run new program and restore saved model and train model from 61 epoch to 120 epoch and save that and close program and repeat this work for your interested epoch