Given a big list of urls, what is a way to check which are active/inactive? [closed]

Answer 1

Perform a HEAD request on each of them.

Use this library: http://docs.python-requests.org/en/latest/user/quickstart/#make-a-request

requests.head('http://httpbin.org/get').status_code

Answer 2

There's an SO answer showing how to perform HEAD requests in Python:

How do you send a HEAD HTTP request in Python 2?

Answer 3

Here is an example in Python

import httplib2

h = httplib2.Http()
listUrls = ['http://www.google.com','http://www.xkcd.com','http://somebadurl.com']
count = 0

for each in listUrls:
    try:
        response, content = h.request(listUrls[count])
        if response.status==200:
            print "UP"
    except httplib2.ServerNotFoundError:
        print "DOWN"
    count = count + 1

Answer 4

Open a pool of threads, open a Url for each, wait for a 200 or a 404. Rinse and repeat.