Training broke with ResourceExausted error

后端 未结 2 1605
梦谈多话
梦谈多话 2020-12-06 06:04

I am new to tensorflow and Machine Learning. Recently I am working on a model. My model is like below,

  1. Character level Embedding Vector -> Embedding lookup

2条回答
  •  囚心锁ツ
    2020-12-06 06:20

    *Note *: The error occurred after 32 epoch. My question is why after 32 epoch there is an error. Why not at the initial epoch.

    This is a major clue that the graph is not static during execution. By that I mean, you're likely doing sess.run(tf.something) instead of

    my_something = tf.something
    with tf.Session() as sess: 
        sess.run(my_something)
    

    I ran into the same problem trying to implement a stateful RNN. I would occasionally reset the state, so I was doing sess.run([reset if some_condition else tf.no_op()]). Simply adding nothing = tf.no_op() to my graph and using sess.run([reset if some_condition else nothing]) solved my problem.

    If you could post the training loop, it would be easier to tell if that is what's going wrong.

提交回复
热议问题