How to create a custom generator class that is correctly garbage collected

问题

I'm trying to write a class in Python that behaves as a generator object, particularly in that when it's garbage collected .close() is called on it. That's important because it means that when the generator is interrupted I can make sure it'll clean up after itself, for example closing files or releasing locks.

Here's some explanatory code: If you interupt a generator, then when it's garbage collected, Python calls .close() on the generator object, which throws a GeneratorExit error into the generator, which can be caught to allow cleanup, like follows:

from threading import Lock

lock = Lock()

def CustomGenerator(n, lock):
    lock.acquire()
    print("Generator Started: I grabbed a lock")
    try:
        for i in range(n):
            yield i
    except GeneratorExit:
        lock.release()
        print("Generator exited early: I let go of the lock")
        raise
    print("Generator finished successfully: I let go of the lock")

for i in CustomGenerator(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Started: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Generator exited early: I let go of the lock
Finished: Lock was free

However, if you try to implement your own generator object by inheriting from collections.abc.Generator, Python doesn't seem to notice that it should call close when the object is collected:

from collections.abc import Generator
class CustomGeneratorClass(Generator):
    def __init__(self, n, lock):
        super().__init__()
        self.lock = lock
        self.lock.acquire()
        print("Generator Class Initialised: I grabbed a lock")
        self.n = n
        self.c = 0

    def send(self, arg):
        value = self.c
        if value >= self.n:
            raise StopIteration
        self.c += 1
        return value

    def throw(self, type, value=None, traceback=None):
        print("Exception Thrown in Generator: I let go of the lock")
        self.lock.release()
        raise StopIteration

for i in CustomGeneratorClass(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Class Initialised: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Oops: Finished, but lock wasn't released

I thought that inheriting Generator would be sufficient to convince python that my CustomGeneratorClass was a generator and should have .close() called on it when garbage collected.

I assume this has something to do with the fact that while 'generator object' are some kind of special Generator:

from types import GeneratorType

c_gen = CustomGenerator(100)
c_gen_class = CustomGeneratorClass(100)

print("CustomGenerator is a Generator:", isinstance(c_gen, Generator))
print("CustomGenerator is a GeneratorType:",isinstance(c_gen, GeneratorType))

print("CustomGeneratorClass is a Generator:",isinstance(c_gen_class, Generator))
print("CustomGeneratorClass is a GeneratorType:",isinstance(c_gen_class, GeneratorType))

CustomGenerator is a Generator: True
CustomGenerator is a GeneratorType: True
CustomGeneratorClass is a Generator: True
CustomGeneratorClass is a GeneratorType: False

Can I make a user defined class object that is GeneratorType?

Is there something I don't understand about how python decides what to call .close() on?

How can I ensure that .close() is called on my custom generator?

This question is not a duplicate of How to write a generator class. For actually making a generator class, the accepted answer for that question does recommends exactly the structure I'm trying here, which is a generator class but is not correctly garbage collected, as shown in the code above.

回答1:

PEP342, states:

[generator].__del__() is a wrapper for [generator].close(). This will be called when the generator object is garbage-collected ...

The Generator class in collections.abc does not implement __del__, and neither do its superclasses or metaclass.

Adding this implementation of __del__ to the class in the question results in the lock being freed:

class CustomGeneratorClass(Generator):

    ...

    def __del__(self):
        self.close()

Output:

Generator Class Initialised: I grabbed a lock
Recieved  0
Recieved  1
Recieved  2
Recieved  3
Exception Thrown in Generator: I let go of the lock
Finished: Lock was free

Caveat:

I'm not experienced with the intricacies of object finalisation in Python, so this suggestion should be examined critically, and tested to destruction. In particular, the warnings about __del__ in the language reference should be considered.

A higher-level solution would be to run the generator in a context manager

with contextlib.closing(CustomGeneratorClass(100, lock)):
    # do stuff

but this is cumbersome, and relies on users of the code remembering to do it.

来源：https://stackoverflow.com/questions/58775283/how-to-create-a-custom-generator-class-that-is-correctly-garbage-collected

标签

python

generator

python-internals