urllib2

Download and decompress gzipped file in memory?

流过昼夜 提交于 2019-11-27 04:02:57
问题 I would like to download a file using urllib and decompress the file in memory before saving. This is what I have right now: response = urllib2.urlopen(baseURL + filename) compressedFile = StringIO.StringIO() compressedFile.write(response.read()) decompressedFile = gzip.GzipFile(fileobj=compressedFile, mode='rb') outfile = open(outFilePath, 'w') outfile.write(decompressedFile.read()) This ends up writing empty files. How can I achieve what I'm after? Updated Answer: #! /usr/bin/env python2

Python-Requests close http connection

跟風遠走 提交于 2019-11-27 03:56:29
I was wondering, how do you close a connection with Requests (python-requests.org)? With httplib it's HTTPConnection.close() , but how do I do the same with Requests? Code is below: r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':toTrack}, auth=('username', 'passwd')) for line in r.iter_lines(): if line: self.mongo['db'].tweets.insert(json.loads(line)) Thanks in advance. Felix Fung As discussed here , there really isn't such a thing as an HTTP connection and what httplib refers to as the HTTPConnection is really the underlying TCP connection which doesn't

Tell urllib2 to use custom DNS

99封情书 提交于 2019-11-27 03:36:20
I'd like to tell urllib2.urlopen (or a custom opener ) to use 127.0.0.1 (or ::1 ) to resolve addresses. I wouldn't change my /etc/resolv.conf , however. One possible solution is to use a tool like dnspython to query addresses and httplib to build a custom url opener. I'd prefer telling urlopen to use a custom nameserver though. Any suggestions? Looks like name resolution is ultimately handled by socket.create_connection . -> urllib2.urlopen -> httplib.HTTPConnection -> socket.create_connection Though once the "Host:" header has been set, you can resolve the host and pass on the IP address

How do I send a POST using 2-legged oauth2 in python?

风格不统一 提交于 2019-11-27 03:06:14
问题 I have a working GET using 2-legged oauth2 in python. Here is the WORKING GET code: the imports: import oauth2 import urllib #for url-encode import urllib2 #for getting and receiving data from server import time #Unix timestamp import oauth2 the call: resourceUrl = "https://test.mysite:8443/ess/scheduleapi/v1/people" request = build_request(resourceUrl,'GET') u = urllib2.urlopen(request.to_url()) people_data = u.read() the function to build the request: def build_request(url, method): params

Python urllib2: Receive JSON response from url

若如初见. 提交于 2019-11-27 02:45:26
I am trying to GET a URL using Python and the response is JSON. However, when I run import urllib2 response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX') html=response.read() print html The html is of type str and I am expecting a JSON. Is there any way I can capture the response as JSON or a python dictionary instead of a str. If the URL is returning valid JSON-encoded data, use the json library to decode that: import urllib2 import json response = urllib2.urlopen('https://api.instagram.com/v1/tags/pizza/media/XXXXXX') data = json.load(response) print data

How to Speed Up Python's urllib2 when doing multiple requests

核能气质少年 提交于 2019-11-27 02:07:52
问题 I am making several http requests to a particular host using python's urllib2 library. Each time a request is made a new tcp and http connection is created which takes a noticeable amount of time. Is there any way to keep the tcp/http connection alive using urllib2? 回答1: If you switch to httplib, you will have finer control over the underlying connection. For example: import httplib conn = httplib.HTTPConnection(url) conn.request('GET', '/foo') r1 = conn.getresponse() r1.read() conn.request(

Unescape Python Strings From HTTP

六月ゝ 毕业季﹏ 提交于 2019-11-27 02:04:12
I've got a string from an HTTP header, but it's been escaped.. what function can I use to unescape it? myemail%40gmail.com -> myemail@gmail.com Would urllib.unquote() be the way to go? I am pretty sure that urllib's unquote is the common way of doing this. >>> import urllib >>> urllib.unquote("myemail%40gmail.com") 'myemail@gmail.com' There's also unquote_plus : Like unquote(), but also replaces plus signs by spaces, as required for unquoting HTML form values. Yes, it appears that urllib.unquote() accomplishes that task. (I tested it against your example on codepad .) Antti Haapala In Python 3

urllib2 not retrieving entire HTTP response

帅比萌擦擦* 提交于 2019-11-27 01:56:03
问题 I'm perplexed as to why I'm not able to download the entire contents of some JSON responses from FriendFeed using urllib2. >>> import urllib2 >>> stream = urllib2.urlopen('http://friendfeed.com/api/room/the-life-scientists/profile?format=json') >>> stream.headers['content-length'] '168928' >>> data = stream.read() >>> len(data) 61058 >>> # We can see here that I did not retrieve the full JSON ... # given that the stream doesn't end with a closing } ... >>> data[-40:] 'ce2-003048343a40","name"

Handling rss redirects with Python/urllib2

非 Y 不嫁゛ 提交于 2019-11-27 01:53:57
问题 Calling urrlib2.urlopen on a link to an article fetched from an RSS feed leads to the following error: urllib2.HTTPError: HTTP Error 301: The HTTP server returned a redirect error tha t would lead to an infinite loop. The last 30x error message was: Moved Permanently According to the documentation, urllib2 supports redirects. On Java the problem was solved by just calling HttpURLConnection.setFollowRedirects(true); How can I solve it with Python? UPDATE The link I'm having problems with: http

How to use python urllib2 to send json data for login

岁酱吖の 提交于 2019-11-27 01:37:28
问题 I want to use python urllib2 to simulate a login action, I use Fiddler to catch the packets and got that the login action is just an ajax request and the username and password is sent as json data, but I have no idea how to use urllib2 to send json data, help... 回答1: import urllib2 import json # Whatever structure you need to send goes here: jdata = json.dumps({"username":"...", "password":"..."}) urllib2.urlopen("http://www.example.com/", jdata) This assumes you're using HTTP POST to send a