Attach a queue to a numpy array in tensorflow for data fetch instead of files?

Question

I have read the CNN Tutorial on the TensorFlow and I am trying to use the same model for my project. The problem is now in data reading. I have around 25000 images for trai

Answer 1

In your question, you specifically asked:

I want to know if I can pass the numpy array as defined above instead of the filenames to some reader and it can fetch records one by one from that array instead of the files.

You can feed the numpy array to a queue directly, but it will be a more invasive change to the cifar10_input.py code than my other answer suggests.

As before, let's assume you have the following array from your question:

import numpy as np
images_and_labels_array = np.array([[...], ...],  # [[1,12,34,24,53,...,102],
                                                  #  [12,112,43,24,52,...,98],
                                                  #  ...]
                                   dtype=np.uint8)

You can then define a queue that contains the entire data as follows:

q = tf.FIFOQueue([tf.uint8, tf.uint8], shapes=[[], [22500]])
enqueue_op = q.enqueue_many([image_and_labels_array[:, 0], image_and_labels_array[:, 1:]])

...then call sess.run(enqueue_op) to populate the queue.

---

Another—more efficient—approach would be to feed records to the queue, which you could do from a parallel thread (see this answer for more details on how this would work):

# [With q as defined above.]
label_input = tf.placeholder(tf.uint8, shape=[])
image_input = tf.placeholder(tf.uint8, shape=[22500])

enqueue_single_from_feed_op = q.enqueue([label_input, image_input])

# Then, to enqueue a single example `i` from the array.
sess.run(enqueue_single_from_feed_op,
         feed_dict={label_input: image_and_labels_array[i, 0],
                    image_input: image_and_labels_array[i, 1:]})

Alternatively, to enqueue a batch at a time, which will be more efficient:

label_batch_input = tf.placeholder(tf.uint8, shape=[None])
image_batch_input = tf.placeholder(tf.uint8, shape=[None, 22500])

enqueue_batch_from_feed_op = q.enqueue([label_batch_input, image_batch_input])

# Then, to enqueue a batch examples `i` through `j-1` from the array.
sess.run(enqueue_single_from_feed_op,
         feed_dict={label_input: image_and_labels_array[i:j, 0],
                    image_input: image_and_labels_array[i:j, 1:]})