how enable requests async mode?

Question

for this code:

import sys

import gevent
from gevent import monkey

monkey.patch_all()

import requests
import urllib2

def worker(url, use_urllib2=False):
          


        

Answer 1

As was pointed out by Kenneth, another thing we can do is to let the requests module handle the asynchronous part. I've made changes to your code accordingly. Again, for me, the results show consistently that requests module performs better than urllib2

Doing this means that we cannot "thread" the call back part. But that should be okay, because the major gain should only be expected with the HTTP requests due to the request/response delay.

import sys

import gevent
from gevent import monkey

monkey.patch_all()

import requests
from requests import async
import urllib2

def call_back(resp):
    content = resp.content
    title = content.split('')[0].strip()
    return title

def worker(url, use_urllib2=False):
    if use_urllib2:
        content = urllib2.urlopen(url).read().lower()
        title = content.split('')[0].strip()

    else:
        rs = [async.get(u) for u in url]
        resps = async.map(rs)
        for resp in resps:
            call_back(resp) 

urls = ['http://www.mail.ru']*5

def by_requests():
    worker(urls)
def by_urllib2():
    jobs = [gevent.spawn(worker, url, True) for url in urls]
    gevent.joinall(jobs)

if __name__=='__main__':
    from timeit import Timer
    t = Timer(stmt="by_requests()", setup="from __main__ import by_requests")
    print 'by requests: %s seconds'%t.timeit(number=3)
    t = Timer(stmt="by_urllib2()", setup="from __main__ import by_urllib2")
    print 'by urllib2: %s seconds'%t.timeit(number=3)
    sys.exit(0)

Here's one of my results:

by requests: 2.44117593765 seconds
by urllib2: 4.41298294067 seconds