How can I speed up fetching pages with urllib2 in python?

Question

I have a script that fetches several web pages and parses the info.

(An example can be seen at http://bluedevilbooks.com/search/?DEPT=MATH&CLASS=103&SEC=01 )

Answer 1

The actual wait is probably not in urllib2 but in the server and/or your network connection to the server.

There are 2 ways of speeding this up.

1. Keep the connection alive (see this question on how to do that: Python urllib2 with keep alive)
2. Use multiplle connections, you can use threads or an async approach as Aaron Gallagher suggested. For that, simply use any threading example and you should do fine :) You can also use the multiprocessing lib to make things pretty easy.