parallelising tf.data.Dataset.from_generator

Question

I have a non trivial input pipeline that from_generator is perfect for...

dataset = tf.data.Dataset.from         


        

Answer 1

Limiting the work done in the generator to a minimum and parallelizing the expensive processing using a map is sensible.

Alternatively, you can "join" multiple generators using parallel_interleave as follows:

def generator(n):
  # returns n-th generator function

def dataset(n):
  return tf.data.Dataset.from_generator(generator(n))

ds = tf.data.Dataset.range(N).apply(tf.contrib.data.parallel_interleave(dataset, cycle_lenght=N))

# where N is the number of generators you use