Tensorflow Enqueue operation was cancelled

问题:

I had built a convolutional neural network in tensorflow. It is trained and now I am unpacking it and performing evaluations.

import main import Process import Input  eval_dir = "/Users/Zanhuang/Desktop/NNP/model.ckpt-250" checkpoint_dir = "/Users/Zanhuang/Desktop/NNP/checkpoint"  def evaluate():   with tf.Graph().as_default() as g:     images, labels = Process.eval_inputs()     forward_propgation_results = Process.forward_propagation(images)     init_op = tf.initialize_all_variables()     saver = tf.train.Saver()     top_k_op = tf.nn.in_top_k(forward_propgation_results, labels, 1)    with tf.Session(graph=g) as sess:     tf.train.start_queue_runners(sess=sess)     sess.run(init_op)     saver.restore(sess, eval_dir)     print(sess.run(top_k_op))   def main(argv=None):     evaluate()  if __name__ == '__main__':   tf.app.run() 

Unfortunately a strange error has popped up and I have no clue why.

W tensorflow/core/kernels/queue_base.cc:2 W tensorflow/core/kernels/queue_base.cc:294] _0_input_producer: Skipping cancelled enqueue attempt with queue not closed W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled      [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]] E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled      [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]] E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled      ....      [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]] E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled      [[Node: batch/fifo_queue_enqueue = QueueEnqueue[Tcomponents=[DT_FLOAT, DT_INT32], _class=["loc:@batch/fifo_queue"], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, Cast_1, Cast)]] W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed ... W tensorflow/core/kernels/queue_base.cc:294] _1_batch/fifo_queue: Skipping cancelled enqueue attempt with queue not closed E tensorflow/core/client/tensor_c_api.cc:485] Enqueue operation was cancelled 

This is only a part of it.

回答1:

Update from chat -- the program runs successfully, and the messages that are printed are due to Python killing threads while they are running as the process exits.

The messages are harmless but it's possible to avoid them by stopping threads manually using pattern below.

coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(sess=sess, coord=coord) <do stuff> coord.request_stop() coord.join(threads) 

回答2:

Everything works correctly and the problem happens at the very last stage when python tries to kill threads. To do this properly you should create a train.Coordinator and pass it to your queue_runner (no need to pass sess, as default session will be used

with tf.Session() as sess:     coord = tf.train.Coordinator()     threads = tf.train.start_queue_runners(coord=coord)     // do your things     coord.request_stop()     coord.join(threads) 

回答3:

a way to add coord with exception handling:

coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(sess, coord)  try:     while not coord.should_stop():          #doing things here  except tf.errors.OutOfRangeError:     print("things done")  finally:     coord.request_stop() 

then bug fixed :-)

文章来源: Tensorflow Enqueue operation was cancelled