How to catch tf.errors.ResourceExhaustedError in tensorflow?

问题

I am using Bayesian regression for hyperparameter optimization, therefore, for some parameters I am running out of memory and this is throwing an error. Therefore, this error could be generated after many iterations, but might not be generated as well. So, I am looking to catch this error and then continue running the code normally. So, I am not sure where to place try and catch block in tensorflow. I guess, the last method throwing the error is session.py. Here is the error:

---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1321     try:
-> 1322       return fn(*args)
   1323     except errors.OpError as e:

...

ResourceExhaustedError: OOM when allocating tensor with shape[14,128,1,788200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/ExpandDims, PermConstNHWCToNCHW-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Any help is much appreciated!!

来源：https://stackoverflow.com/questions/52392502/how-to-catch-tf-errors-resourceexhaustederror-in-tensorflow