StopIteration: generator_output = next(output_generator)

Question

I have the following code which I rewrite to work on a large scale dataset. I am using Python generator to Fit the model on data yielded batch-by-batch.

def          


        

Answer 1

Generators for keras must be infinite:

def subtract_mean_gen(x_source,y_source,avg_image,batch):
    while True:
        batch_list_x=[]
        batch_list_y=[]
        for line,y in zip(x_source,y_source):
            x=line.astype('float32')
            x=x-avg_image
            batch_list_x.append(x)
            batch_list_y.append(y)
            if len(batch_list_x) == batch:
                yield (np.array(batch_list_x),np.array(batch_list_y))
                batch_list_x=[]
                batch_list_y=[] 

The error happens because keras tries to get a new batch, but your generator has already reached its end. (Even though you defined a correct number of steps, keras has a queue that will be trying to get more batches from the generator even if you are at the last step.)

Apparently, you've got a default queue size, which is 10 (the exception appears 10 batches before the end because the queue is trying to get a batch after the end).