Convert a curl POST request to Python only using standard library

Question

I would like to convert this curl command to something that I can use in Python for an existing script.

curl -u 7898678:X -H \'Content-Type: application/jso         


        

Answer 1

While there are ways to handle authentication in urllib2, if you're doing Basic Authorization (which means effectively sending the username and password in clear text) then you can do all of what you want with a urllib2.Request and urllib2.urlopen:

import urllib2

def basic_authorization(user, password):
    s = user + ":" + password
    return "Basic " + s.encode("base64").rstrip()

req = urllib2.Request("http://localhost:8000/36576/speak.json",
                      headers = {
        "Authorization": basic_authorization("7898678", "X"),
        "Content-Type": "application/json",

        # Some extra headers for fun
        "Accept": "*/*",   # curl does this
        "User-Agent": "my-python-app/1", # otherwise it uses "Python-urllib/..."
        },
                      data = '{"message":{"body":"TEXT"}}')

f = urllib2.urlopen(req)

I tested this with netcat so I could see that the data sent was, excepting sort order, identical in both cases. Here the first one was done with curl and the second with urllib2

% nc -l 8000
POST /36576/speak.json HTTP/1.1
Authorization: Basic Nzg5ODY3ODpY
User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 27

{"message":{"body":"TEXT"}} ^C

% nc -l 8000
POST /36576/speak.json HTTP/1.1
Accept-Encoding: identity
Content-Length: 27
Connection: close
Accept: */*
User-Agent: my-python-app/1
Host: localhost:8000
Content-Type: application/json
Authorization: Nzg5ODY3ODpY

{"message":{"body":"TEXT"}}^C

(This is slightly tweaked from the output. My test case didn't use the same url path you used.)

There's no need to use the underlying httplib, which doesn't support things that urllib2 gives you like proxy support. On the other hand, I do find urllib2 to be complicated outside of this simple sort of request and if you want better support for which headers are sent and the order they are sent then use httplib.

Answer 2

Take a look at pycurl http://pycurl.sourceforge.net/

Answer 3

Thanks every

this works

import urllib2

def speak(status):

    def basic_authorization(user, password):
        s = user + ":" + password
        return "Basic " + s.encode("base64").rstrip()

    req = urllib2.Request("http://example.com/60/speak.json",
                  headers = {
       "Authorization": basic_authorization("2345670", "X"),
     "Content-Type": "application/json",


        "Accept": "*/*",   
        "User-Agent": "my-python-app/1", 
        },
                      data = '{"message":{"body":'+ status +'}}')

    f = urllib2.urlopen(req)


speak('Yay')

Answer 4

I would like this to work with the standard library if possible.

The standard library provides urllib and httplib for working with URLs:

>>> import httplib, urllib
>>> params = urllib.urlencode({'apple': 1, 'banana': 2, 'coconut': 'yummy'})
>>> headers = {"Content-type": "application/x-www-form-urlencoded",
...            "Accept": "text/plain"}
>>> conn = httplib.HTTPConnection("example.com:80")
>>> conn.request("POST", "/some/path/to/site", params, headers)
>>> response = conn.getresponse()
>>> print response.status, response.reason
200 OK

If you want to execute curl itself, though, you can just invoke os.system():

import os
TEXT = ...
cmd = """curl -u 7898678:X -H 'Content-Type: application/json'""" \
  """-d '{"message":{"body":"%{t}"}}' http://sample.com/36576/speak.json""" % \
  {'t': TEXT}

If you're willing to relax the standard-library-only restriction, you can use PycURL. Beware that it isn't very Pythonic (it's pretty much just a thin veneer over libcurl), and I'm not sure how compatible it is with Python 3.