Python Requests Multipart HTTP POST

Question

I was wondering how do you translate something like this using Python Requests? In urllib2, you can manually manipulate the data that is being sent over the wire to the API service, but Requests claims multipart file uploads are easy. However, when trying to send over the same request using the Requests library, I believe that it is not specifying some key parameters in the content-type for each of the two parts correctly. Can someone please shed some light on this matter. Thank you in advance!

def upload_creative(self, account_id, file_path):
    """"""
    boundary = '-----------------------------' + str(int(random.random()*1e10))
    parts = []

    # Set account ID part.
    parts.append('--' + boundary)
    parts.append('Content-Disposition: form-data; name="account_id"')
    parts.append('')
    parts.append(str(account_id))

    # Set creative contents part.
    parts.append('--' + boundary)
    parts.append('Content-Disposition: form-data; name="userfile"; filename="%s"' % file_path)
    parts.append('Content-Type: %s' % mimetypes.guess_type(file_path)[0] or 'application/octet-stream')
    parts.append('')
    # TODO: catch errors with opening file.
    parts.append(open(file_path, 'r').read())

    parts.append('--' + boundary + '--')
    parts.append('')

    body = '\r\n'.join(parts)

    headers = {'content-type': 'multipart/form-data; boundary=' + boundary}
    url = self._resolve_url('/a/creative/uploadcreative')
    req = urllib2.Request(url, headers=headers, data=body)
    res = urllib2.urlopen(req)

    return json.loads(res.read())

When I examine firebug from the UI, I get the following in the POST source.

-----------------------------662549079759661058833120391 
Content-Disposition: form-data; name="userfile"; filename="IMG_1377.jpg" Content-Type: image/jpeg 

ÿØÿáÃExif��MM�*���� �������ª���� ���°���������������������º�������Â(�������1�������Ê2�������Ú<�������î�������i�������þ%������p��Apple�iPhone 4���H������H�����QuickTime 7.7.1�2012:08:17 11:47:11�Mac OS X 10.7.4�������������� "�������'�����P�������0220������(������<������ �����P������X������� ����� �� ������`������h �����0100 ������� ������  ������¢�������¤��������¤��������¤��������¤ ����������������������2011:10:01 17:19:23�2011:10:01 17:19:23���4��Á��¹��¡���M���Ç»¸������N����������Ê�����W����������â�������ú�����M�����������������!�����S���d����������T�����ÿ���d���������������������Ú����%Á��r��������������t������|(�������������������7������������H������H�����ÿØÿà�JFIF��H�H��ÿþ�AppleMark ÿÛ��  % #!,!#'(***.1-)1%)*( (((((((((((((((((((((((((((((((((((((((((((((((((((ÿÄ¢���������� ������� ���}�!1AQa"q2¡#B±ÁRÑð$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz¢£¤¥¦§¨©ª²³´µ¶·¸¹ºÂÃÄÅÆÇÈÉÊÒÓÔÕÖ×ØÙÚáâãäåæçèéêñòóôõö÷øùú��w�!1AQaq"2B¡±Á #3RðbrÑ $4á%ñ&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz¢£¤¥¦§¨©ª²³´µ¶·¸¹ð.¥ÛWíÇLòV³FcaoæÂÒ8§É¸(è3E¢"Ú×S^+yj�!òû0Oüµn- yè){[oÝ/¸?ÃÔMY¡ÃgÔò4êò4n͸í¶={ÔM¤¸m¯K&ñæ«,©ù»zTÝ=öØô×6¶Ö:MÑi�,Û$Oö[ª÷ª©ÆiîỤJAxj>ÞAõúu¥}lIf÷û^Â)#´y^)Ô"/·v>n~4ººµ­¬æ}FURì·Î 3¿Ãèh»ÐµÈÿ�·|Gu:ß²<ëlWäG·^+¡Ó¼gâ.-Þè|ϸ*ª®  }é?Ú=(i:2½Ïg!ʵÑi¤¼eþ!÷³ÍC'æCqv®ÖÊÕiçCë·øsQy#K_B´þ0s'¦|¿Þ²lò¼?½ÿ�]rZ¶¨ø·6ñÆØ·mvV;þÿ�þ=ôª¿»r\zPñtHö÷>Ù¤R#+ Á òBôR;ú²¾)!àËn<.ÁÔlÏcRäÂ&§­eX´fTóLžQßt§Zµ{â t·pK]ÈL1²îýúEüxþ÷j\î×-jÏÂ>!û:^,E­,>^ýêßwû+Ópæ»?i÷û5kéá¹^ 6Ddq°öÁ¯Rù¨¦yãjòÿÙ 
-----------------------------662549079759661058833120391 
Content-Disposition: form-data; name="account_id" 

69574 
-----------------------------662549079759661058833120391--

The headers in firebug are as follows:

Request Headersview source
Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-us,en;q=0.5
Cache-Control   no-cache
Connection  keep-alive
Content-Length  1713991
Content-Type    multipart/form-data; boundary=---------------------------662549079759661058833120391
Cookie  instance_defaults=%7C%20%7Cen_US; access_token=75c48e
Host    ui.host.com
Pragma  no-cache
Referer http://ui.host.com/
User-Agent  Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:14.0) Gecko/20100101 Firefox/14.0.1

I guess my question is is there any way via the requests library to adjust the data so that the:

Content-Disposition: form-data; name="userfile"; filename="IMG_1377.jpg" Content-Type: image/jpeg

and the

Content-Disposition: form-data; name="account_id" 

69574

statements are both present. I feel that I would have to do something like have files be a dictionary of

files = {'file': open('image.jpg', 'rb'), 'account_id': 12345}

but somehow edit the Content-Disposition metadata of each of these parts separately

Answer 1

with requests, I believe that you don't have to be so manual, simply:

import requests

# ...
url = self._resolve_url('/a/creative/uploadcreative')
files = {'file': ('userfile', open(filepath, 'rb'))}
data = {'account_id': account_id}
headers = {'content-type': 'multipart/form-data'}
res = requests.post(url, files=files, data=data, headers=headers)
return res.json

I suppose your concern lies with your:

parts.append('Content-Type: %s' % mimetypes.guess_type(file_path)[0] or 'application/octet-stream')

I haven't proven it to myself beyond the shadow of a doubt. But, I think that is built in to requests here.

Edit: It looks like you can have the normal fields in the files dict, as you propose:

files = {'file': open('image.jpg', 'rb'), 'account_id': 12345}

and could name the filename as you want:

files = {'file': ('userfile', open('image.jpg', 'rb')), 'account_id': 12345}

but, you would get a body.write(b'Content-Type: text/plain\r\n\r\n') on the account_id field which is probably not what you want and have no way to customize the Content-Disposition for each field (still not sure why you would need to); for both the file and the field you will get: Content-Disposition: form-data - which is what you show for both.

I'm not sure that you can do exactly what you want with requests, maybe you should try a feature request.

Answer 2

 import requests

 import urllib

 def upload_creative(self, account_id, file_path):

    files = [('userfile', (file_path, open(file_path, 'rb'), "image/jpeg" ))]

    url = self._resolve_url('/a/creative/uploadcreative')

    url =  url + "?"  +  urlib.urlencode(account_id=account_id)

    reuests.post(url, files=files)

Answer 3

I found out that in the python-requests library (v.0.13.3), your data will get wiped if you include the "data" field before the "files" field in the request call itself.

For example,

requests.post(url, headers=headers, data=data, files=files) 

will yield empty form-data. However, the following will send the data dictionary as form-data

requests.post(url, headers=headers, files=files, data=data)

Thanks everyone for their answers