tensorflow-datasets

I am setting up a TensorFlow pipeline for reading large HDF5 files as input for my deep learning models. Each HDF5 file contains 100 videos of variable size length stored as a collection of compressed JPG images (to make size on disk manageable). Using tf.data.Dataset and a map to tf.py_func , reading examples from the HDF5 file using custom Python logic is quite easy. For example: def read_examples_hdf5(filename, label): with h5py.File(filename, 'r') as hf: # read frames from HDF5 and decode them from JPG return frames, label filenames = glob.glob(os.path.join(hdf5_data_path, "*.h5")) labels

For transfer learning, one often uses a network as a feature extractor to create a dataset of features, on which another classifier is trained (e.g. a SVM). I want to implement this using the Dataset API ( tf.contrib.data ) and dataset.map() : # feature_extractor will create a CNN on top of the given tensor def features(feature_extractor, ...): dataset = inputs(...) # This creates a dataset of (image, label) pairs def map_example(image, label): features = feature_extractor(image, trainable=False) # Leaving out initialization from a checkpoint here... return features, label dataset = dataset

In the tensorflow Dataset pipeline I'd like to define a custom map function which takes a single input element (data sample) and returns multiple elements (data samples). The code below is my attempt, along with the desired results. I could not follow the documentation on tf.data.Dataset().flat_map() well enough to understand if it was applicable here or not. import tensorflow as tf input = [10, 20, 30] def my_map_func(i): return [[i, i+1, i+2]] # Fyi [[i], [i+1], [i+2]] throws an exception ds = tf.data.Dataset.from_tensor_slices(input) ds = ds.map(map_func=lambda input: tf.py_func( func=my

TF Map function supports parallel calls . I'm seeing no improvements passing num_parallel_calls to map. With num_parallel_calls=1 and num_parallel_calls=10 , there is no improvement in performance run time. Here is a simple code import time def test_two_custom_function_parallelism(num_parallel_calls=1, batch=False, batch_size=1, repeat=1, num_iterations=10): tf.reset_default_graph() start = time.time() dataset_x = tf.data.Dataset.range(1000).map(lambda x: tf.py_func( squarer, [x], [tf.int64]), num_parallel_calls=num_parallel_calls).repeat(repeat) if batch: dataset_x = dataset_x.batch(batch