Singleton python generator? Or, pickle a python generator?

Question

I am using the following code, with nested generators, to iterate over a text document and return training examples using get_train_minibatch(). I would like to

Answer 1

You can create a standard iterator object, it just won't be as convenient as the generator; you need to store the iterator's state on the instace (so that it is pickled), and define a next() function to return the next object:

class TrainExampleIterator (object):
    def __init__(self):
        # set up internal state here
        pass
    def next(self):
        # return next item here
        pass

The iterator protocol is simple as that, defining the .next() method on an object is all you need to pass it to for loops etc.

In Python 3, the iterator protocol uses the __next__ method instead (somewhat more consistent).