Why does this contextmanager behave differently with dict comprehensions?

问题

I have a context decorator that has side effects when it's done. I've noticed that the side effects don't occur if I use a dict comprehension.

from contextlib import contextmanager
import traceback
import sys

accumulated = []

@contextmanager
def accumulate(s):
    try:
        yield
    finally:
        print("Appending %r to accumulated" % s)
        accumulated.append(s)

def iterate_and_accumulate(iterable):
    for item in iterable:
        with accumulate(item):
            yield item

def boom_unless_zero(i):
    if i > 0:
        raise RuntimeError("Boom!")

try:
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
except:
    traceback.print_exc()

print(accumulated)

print('\n=====\n')

try:
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
except:
    traceback.print_exc()

print(accumulated)
print('Finished!')

Output:

$ python2 boom3.py 
Appending 0 to accumulated
Traceback (most recent call last):
  File "boom3.py", line 25, in <module>
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
  File "boom3.py", line 25, in <dictcomp>
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
  File "boom3.py", line 22, in boom_unless_zero
    raise RuntimeError("Boom!")
RuntimeError: Boom!
[0]

=====

Appending 0 to accumulated
Appending 1 to accumulated
Traceback (most recent call last):
  File "boom3.py", line 34, in <module>
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
  File "boom3.py", line 34, in <dictcomp>
    {i: boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])}
  File "boom3.py", line 22, in boom_unless_zero
    raise RuntimeError("Boom!")
RuntimeError: Boom!
[0, 0, 1]
Finished!
Appending 1 to accumulated

It's bizarre that the side effect occurs after my script is 'finished'. It means users can't use my contextdecorator if they're using dict comprehensions.

I've noticed that this behaviour disappears on Python 3, and the behaviour also doesn't occur if I write [boom_unless_zero(i) for i in iterate_and_accumulate([0, 1])] instead of the dict comprehension.

Why does this occur?

回答1:

From https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement:

As of Python version 2.5, the yield statement is now allowed in the try clause of a try ... finally construct. If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator’s close() method will be called, allowing any pending finally clauses to execute.

In other words, pending finally clauses will not execute until the generator-iterator is closed, either explicitly or as a result of it being garbage-collected (refcount or cyclic). It seems that Python 2 list comprehensions and Python 3 are more efficient at garbage collecting the iterable.

If you want to be explicit about closing the generator-iterator:

from contextlib import closing

try:
    with closing(iter(iterate_and_accumulate(a))) as it:
        {i: boom_unless_zero(i) for i in it}
except:
    traceback.print_exc()
print(accumulated)

I had a look at the underlying issue; it seems that the problem is that the generator-iterator is held by the exception traceback state, so another workaround is to call sys.exc_clear():

import sys

try:
    {i: boom_unless_zero(i) for i in iterate_and_accumulate(a)}
except:
    traceback.print_exc()
    try:
        sys.exc_clear()
    except AttributeError:
        pass
print(accumulated)

In Python 3, the lexical exception handler system (http://bugs.python.org/issue3021) means that the exception state is cleared on exit from the handler block, so sys.exc_clear() is not necessary (and indeed is not present).

来源：https://stackoverflow.com/questions/24507404/why-does-this-contextmanager-behave-differently-with-dict-comprehensions

标签

python

python-2.7

exception

generator