Load data from Python pickle file in a loop?

问题

In a small data-acquisition project we use the Python's pickle to store recorded data, i.e. for each "event" we add it to the output file f with

pkl.dump(event, f, pkl.HIGHEST_PROTOCOL)

where import cPickle as pkl.

In the analysis of the data we read each event, but in contrast to a normal file where processing can be one in a rather elegant way:

with open(filename) as f:
    for line in f:
        do_something()

looping over all the data in a pickle file this becomes a bit more awkward:

with open(filename) as f:
    try:
        while True:
            event = pkl.load(f)
            do_something()
    except (EOFError, UnpicklingError):
        pass

Is it possible to make pickle reading more like the example for regular files above?

回答1:

Yes, indeed. Use this generator below to make the events readable in a loop:

def pickleLoader(pklFile):
    try:
        while True:
            yield pkl.load(pklFile)
    except EOFError:
        pass

Now you can simply write:

with open(filename) as f:
    for event in pickleLoader(f):
        do_something()

来源：https://stackoverflow.com/questions/18675863/load-data-from-python-pickle-file-in-a-loop