I am new to gevents and greenlets. I found some good documentation on how to work with them, but none gave me justification on how and when I should use greenlets!
Taking @Max's answer and adding some relevance to it for scaling, you can see the difference. I achieved this by changing the URLs to be filled as follows:
URLS_base = ['www.google.com', 'www.example.com', 'www.python.org', 'www.yahoo.com', 'www.ubc.ca', 'www.wikipedia.org']
URLS = []
for _ in range(10000):
for url in URLS_base:
URLS.append(url)
I had to drop out the multiprocess version as it fell before I had 500; but at 10,000 iterations:
Using gevent it took: 3.756914
-----------
Using multi-threading it took: 15.797028
So you can see there is some significant difference in I/O using gevent