Windows Authentication with Python and urllib2

Question

I want to grab some data off a webpage that requires my windows username and password.

So far, I've got:

opener = build_opener()
try:
    page = opener.open("http://somepagewhichneedsmywindowsusernameandpassword/")
    print page
except URLError:
    print "Oh noes."

Is this supported by urllib2? I've found Python NTLM, but that requires me to put my username and password in. Is there any way to just grab the authentication information somehow (e.g. like IE does, or Firefox, if I changed the network.automatic-ntlm-auth.trusted-uris settings).

Edit after msander's answer

So I've now got this:

# Send a simple "message" over a socket - send the number of bytes first,
# then the string.  Ditto for receive.
def _send_msg(s, m):
    s.send(struct.pack("i", len(m)))
    s.send(m)

def _get_msg(s):
    size_data = s.recv(struct.calcsize("i"))
    if not size_data:
        return None
    cb = struct.unpack("i", size_data)[0]
    return s.recv(cb)

def sspi_client():
    c = httplib.HTTPConnection("myserver")
    c.connect()
    # Do the auth dance.
    ca = sspi.ClientAuth("NTLM", win32api.GetUserName())
    data = None
    while 1:
        err, out_buf = ca.authorize(data) # error 400 triggered by this line
        _send_msg(c.sock, out_buf[0].Buffer)

        if err==0:
            break

        data = _get_msg(c.sock)

    print "Auth dance complete - sending a few encryted messages"
    # Assume out data is sensitive - encrypt the message.
    for data in "Hello from the client".split():
        blob, key = ca.encrypt(data)
        _send_msg(c.sock, blob)
        _send_msg(c.sock, key)
    c.sock.close()
    print "Client completed."

which is pretty well ripped from socket_server.py (see here). But I get an error 400 - bad request. Does anyone have any further ideas?

Thanks,

Dom

Answer 1

Assuming you are writing your client code on Windows and need seamless NTLM authentication then you should read Mark Hammond's Hooking in NTLM post from the python-win32 mailing list which essentially answers the same question. This points at the sspi example code included with the Python Win32 extensions (which are included with ActivePython and otherwise can be downloaded here).

Answer 2

There are several forms of authentication that web sites can use.

1. HTTP Authentication. This where the browser pops up a window for you to enter your username and password. There are two mechanisms: basic and digest. There is an "Authorization" Header that comes along with the page that tells a browser (or a program using urllib2) what to do.

   In this case, you must configure your urlopener to provide the answers that the authorization header needs to see. You'll need to build either an HTTPBasicAuthHandler or HTTPDigestAuthHandler.

   AuthHandlers require a PasswordManager. This password manager could have a hard-coded username and password (very common) or it could be clever and work out your Windows password from some Windows API.

2. Application Authentication. This is where the web application directs you to a page with a form you fill in with a username and password. In this case, your Python program must use urllib2 to do a POST (a request with data) where the data is the form filled in properly. The reply to the post usually contains a cookie, which is what allows you further access. You don't need to worry much about the cookie, urllib2 handles this automatically.

How do you know which you have? You dump the headers on the response. The response from urllib2.openurl includes all the headers (in page.info()) as well as the page content.

Read HTTP Authentication in Python

How would one log into a phpBB3 forum through a Python script using urllib, urllib2 and ClientCookie?

How do you access an authenticated Google App Engine service from a (non-web) python client?