urllib

How to send POST request?

由 孤人 提交于 2019-11-26 02:16:53
问题 I found this script online: import httplib, urllib params = urllib.urlencode({\'number\': 12524, \'type\': \'issue\', \'action\': \'show\'}) headers = {\"Content-type\": \"application/x-www-form-urlencoded\", \"Accept\": \"text/plain\"} conn = httplib.HTTPConnection(\"bugs.python.org\") conn.request(\"POST\", \"\", params, headers) response = conn.getresponse() print response.status, response.reason 302 Found data = response.read() data \'Redirecting to <a href=\"http://bugs.python.org

How to percent-encode URL parameters in Python?

由 心不动则不痛 提交于 2019-11-26 01:56:27
问题 If I do url = \"http://example.com?p=\" + urllib.quote(query) It doesn\'t encode / to %2F (breaks OAuth normalization) It doesn\'t handle Unicode (it throws an exception) Is there a better library? 回答1: From the docs: urllib.quote(string[, safe]) Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-' are never quoted. By default, this function is intended for quoting the path section of the URL.The optional safe parameter specifies additional

UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode characters

由 江枫思渺然 提交于 2019-11-26 01:35:44
问题 I\'m trying to scrape a website, but it gives me an error. I\'m using the following code: import urllib.request from bs4 import BeautifulSoup get = urllib.request.urlopen(\"https://www.website.com/\") html = get.read() soup = BeautifulSoup(html) print(soup) And I\'m getting the following error: File \"C:\\Python34\\lib\\encodings\\cp1252.py\", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: \'charmap\' codec can\'t encode characters in

can we use xpath with BeautifulSoup?

由 我们两清 提交于 2019-11-26 01:05:30
I am using BeautifulSoup to scrape a url and I had the following code import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = "http://www.example.com/servlet/av/ResultTemplate=AVResult.html" req = urllib2.Request(url) response = urllib2.urlopen(req) the_page = response.read() soup = BeautifulSoup(the_page) soup.findAll('td',attrs={'class':'empformbody'}) Now in the above code we can use findAll to get tags and information related to them, but I want to use xpath. Is it possible to use xpath with BeautifulSoup? If possible, can anyone please provide me an example code so that

can we use xpath with BeautifulSoup?

由 二次信任 提交于 2019-11-26 00:58:34
问题 I am using BeautifulSoup to scrape a url and I had the following code import urllib import urllib2 from BeautifulSoup import BeautifulSoup url = \"http://www.example.com/servlet/av/ResultTemplate=AVResult.html\" req = urllib2.Request(url) response = urllib2.urlopen(req) the_page = response.read() soup = BeautifulSoup(the_page) soup.findAll(\'td\',attrs={\'class\':\'empformbody\'}) Now in the above code we can use findAll to get tags and information related to them, but I want to use xpath. Is

What are the differences between the urllib, urllib2, urllib3 and requests module?

由 依然范特西╮ 提交于 2019-11-26 00:07:16
问题 In Python, what are the differences between the urllib , urllib2 , urllib3 and requests module? Why are there three? They seem to do the same thing... 回答1: I know it's been said already, but I'd highly recommend the requests Python package. If you've used languages other than python, you're probably thinking urllib and urllib2 are easy to use, not much code, and highly capable, that's how I used to think. But the requests package is so unbelievably useful and short that everyone should be

How do I download a file over HTTP using Python?

由 心已入冬 提交于 2019-11-25 23:57:12
问题 I have a small utility that I use to download a MP3 from a website on a schedule and then builds/updates a podcast XML file which I\'ve obviously added to iTunes. The text processing that creates/updates the XML file is written in Python. I use wget inside a Windows .bat file to download the actual MP3 however. I would prefer to have the entire utility written in Python though. I struggled though to find a way to actually down load the file in Python, thus why I resorted to wget . So, how do

Downloading a picture via urllib and python

由 生来就可爱ヽ(ⅴ＜●) 提交于 2019-11-25 23:39:51
问题 So I\'m trying to make a Python script that downloads webcomics and puts them in a folder on my desktop. I\'ve found a few similar programs on here that do something similar, but nothing quite like what I need. The one that I found most similar is right here (http://bytes.com/topic/python/answers/850927-problem-using-urllib-download-images). I tried using this code: >>> import urllib >>> image = urllib.URLopener() >>> image.retrieve(\"http://www.gunnerkrigg.com//comics/00000001.jpg\",\

urllib and “SSL: CERTIFICATE_VERIFY_FAILED” Error

由 房东的猫 提交于 2019-11-25 21:53:52
问题 I am getting the following error: Exception in thread Thread-3: Traceback (most recent call last): File \"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py\", line 810, in __bootstrap_inner self.run() File \"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py\", line 763, in run self.__target(*self.__args, **self.__kwargs) File \"/Users/Matthew/Desktop/Skypebot 2.0/bot.py\", line 271, in process info = urllib2.urlopen(req).read() File \"

UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode characters

由 ☆樱花仙子☆ 提交于 2019-11-25 19:04:15
I'm trying to scrape a website, but it gives me an error. I'm using the following code: import urllib.request from bs4 import BeautifulSoup get = urllib.request.urlopen("https://www.website.com/") html = get.read() soup = BeautifulSoup(html) print(soup) And I'm getting the following error: File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode characters in position 70924-70950: character maps to <undefined> What can I do to fix this? I fixed it by adding .encode("utf-8")
订阅 urllib