Non-blocking generator on Python

问题

I'm using a generator function from the requests module in a QT-Application, pretty much the same as in the requests-streaming example:

import json
import requests

def get_stream():
    r = requests.get('http://httpbin.org/stream/20', stream=True)
    for line in r.iter_lines():
        if line:
            yield json.loads(line)

def consume_stream():
    for message in get_stream():
       #do something

However, when there is no incoming response (f.e. irregularly incoming tweets from Twitters Streaming API), the generator get_stream will block the consume_stream method.

This might occur in any situation where a generator does not yield immediately, but hast to wait for incoming messages etc., and therefore blocks the consumer.

Is there any pattern in Python where you can consume a generator in a non-blocking way, i.e. if the generator yields, process it's results, otherwise do something else until the next results are occuring?

回答1:

Take a look at the producer-consumer pattern. It is commonly implemented in python using a Queue.

The producer, typically running in a thread or another process (Queue supports either), simply puts messages in the queue. The consumer, whenever it feels like it, pops messages from the queue. This operation supports a timeout argument.

回答2:

As Simeon is asking in the comment, it cannot work as simple as you describe it in your example. There are quite some details you have to take care of. There are different solutions, which make more or less sense, depending on your use case. You don't give much details about what you really want to do, so I will just send you to http://twistedmatrix.com/trac/wiki/QTReactor as an example. There a different solutions/frameworks which implement async message queues. And I think, that's what you are looking for.

回答3:

If you control the generator function, one solution would be to have it throw an exception after a timeout period. Perhaps something along the lines of:

def get_stream(timeout=None):
    while message=read_message(timeout=timout):
        yield message

Then have read_message throw a TimeOutException or something if a timeout condition occurs.

Of course, you still have to deal with the logistics of when/how to retry/resume.

来源：https://stackoverflow.com/questions/19736463/non-blocking-generator-on-python

标签

python

generator

yield