urllib2

I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module. I used the inbuilt 2to3 Python tool to convert the below working urllib and urllib2 Python 2.7 code: import urllib2 import urllib url = "https://www.customdomain.com" d = dict(parameter1="value1", parameter2="value2") req = urllib2.Request(url, data=urllib.urlencode(d)) f = urllib2.urlopen(req) resp = f.read() The output from the 2to3 module was the below Python 3 code: import urllib.request, urllib.error, urllib.parse url = "https://www.customdomain.com" d = dict

I need to set the timeout on urllib2.request() . I do not use urllib2.urlopen() since i am using the data parameter of request . How can I set this? Although urlopen does accept data param for POST , you can call urlopen on a Request object like this, import urllib2 request = urllib2.Request('http://www.example.com', data) response = urllib2.urlopen(request, timeout=4) content = response.read() Giorgoc still, you can avoid using urlopen and proceed like this: request = urllib2.Request('http://example.com') response = opener.open(request,timeout=4) response_result = response.read() this works

I am using following code to save webpage using Python: import urllib import sys from bs4 import BeautifulSoup url = 'http://www.vodafone.de/privat/tarife/red-smartphone-tarife.html' f = urllib.urlretrieve(url,'test.html') Problem : This code saves html as basic html without javascripts, images etc. I want to save webpage as complete (Like we have option in browser) Update : I am using following code now to save all the js/images/css files of webapge so that it can be saved as complete webpage but still my output html is getting saved like basic html: import pycurl import StringIO c = pycurl

Is it possible to fetch pages with urllib2 through a SOCKS proxy on a one socks server per opener basic? I've seen the solution using setdefaultproxy method, but I need to have different socks in different openers. So there is SocksiPy library, which works great, but it has to be used this way: import socks import socket socket.socket = socks.socksocket import urllib2 socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "x.x.x.x", y) That is, it sets the same proxy for ALL urllib2 requests. How can I have different proxies for different openers? Try with pycurl : import pycurl c1 = pycurl.Curl() c1

I have a piece of code like this host = 'http://www.bing.com/search?q=%s&go=&qs=n&sk=&sc=8-13&first=%s' % (query, page) req = urllib2.Request(host) req.add_header('User-Agent', User_Agent) response = urllib2.urlopen(req) and when I input a query greater than one word like "the dog" i get the following error. response = urllib2.urlopen(req) File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 400, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 513, in http_response 'http',

I need to do a API call to upload a file along with a JSON string with details about the file. I am trying to use the python requests lib to do this: import requests info = { 'var1' : 'this', 'var2' : 'that', } data = json.dumps({ 'token' : auth_token, 'info' : info, }) headers = {'Content-type': 'multipart/form-data'} files = {'document': open('file_name.pdf', 'rb')} r = requests.post(url, files=files, data=data, headers=headers) This throws the following error: raise ValueError("Data must not be a string.") ValueError: Data must not be a string If I remove the 'files' from the request, it

PHP code: <?php $data=$_POST['data']; echo $data; ?> When I do that, the HTML page that Python prints notifies me that PHP did not receive any value in $data I.e: Error in $name; undefined index However, when I send the data as GET ( http://localhost/mine.php?data=data ) and change the PHP method from POST to GET ( $data=$_GET['data'] ), the value is gotten and processed. My main issue here is that it seems the value in data does not go through to PHP as I would have wanted to use POST. What could be wrong? Look at this python: import urllib2, urllib mydata=[('one','1'),('two','2')] #The first