urllib2

They didn't mention this in python documentation. And recently I'm testing a website simply refreshing the site using urllib2.urlopen() to extract certain content, I notice sometimes when I update the site urllib2.urlopen() seems not get the newly added content. So I wonder it does cache stuff somewhere, right? So I wonder it does cache stuff somewhere, right? It doesn't. If you don't see new data, this could have many reasons. Most bigger web services use server-side caching for performance reasons, for example using caching proxies like Varnish and Squid or application-level caching. If the

Here is the code: import urllib2 as URL def get_unread_msgs(user, passwd): auth = URL.HTTPBasicAuthHandler() auth.add_password( realm='New mail feed', uri='https://mail.google.com', user='%s'%user, passwd=passwd ) opener = URL.build_opener(auth) URL.install_opener(opener) try: feed= URL.urlopen('https://mail.google.com/mail/feed/atom') return feed.read() except: return None It works just fine. The only problem is that when a wrong username or password is used, it takes forever to open to url @ feed= URL.urlopen('https://mail.google.com/mail/feed/atom') It doesn't throw up any errors, just keep

I'm trying to learn python, so I decided to write a script that could translate something using google translate. Till now I wrote this: import sys from BeautifulSoup import BeautifulSoup import urllib2 import urllib data = {'sl':'en','tl':'it','text':'word'} request = urllib2.Request('http://www.translate.google.com', urllib.urlencode(data)) request.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11') opener = urllib2.build_opener() feeddata = opener.open(request).read() #print feeddata soup = BeautifulSoup(feeddata) print soup

I am programming a software in Python to download HTTP PDF from a database. Sometimes the download stop with this message : retrieval incomplete: got only 3617232 out of 10689634 bytes How can I ask the download to restart where it stops using the 206 Partial Content HTTP feature ? I can do it using wget -c and it works pretty well, but I would like to implement it directly in my Python software. Any idea ? Thank you You can request a partial download by sending a GET with the Range header: import urllib2 req = urllib2.Request('http://www.python.org/') # # Here we request that bytes 18000-

I am not able to open one particular url using urllib2. Same approach works well with other websites such as "http://www.google.com" but not this site (which also displays fine in the browser). my simple code: from BeautifulSoup import BeautifulSoup import urllib2 url="http://www.experts.scival.com/einstein/" response=urllib2.urlopen(url) html=response.read() soup=BeautifulSoup(html) print soup Can anyone help me to make it work? this is error I got: Traceback (most recent call last): File "/Users/jontaotao/Documents/workspace/MedicalSchoolInfo/src/AlbertEinsteinCollegeOfMedicine_SciValExperts

This work fine: import urllib2 opener = urllib2.build_opener( urllib2.HTTPHandler(), urllib2.HTTPSHandler(), urllib2.ProxyHandler({'http': 'http://user:pass@proxy:3128'})) urllib2.install_opener(opener) print urllib2.urlopen('http://www.google.com').read() But, if http change to https : ... print urllib2.urlopen('https://www.google.com').read() There are errors: Traceback (most recent call last): File "D:\Temp\6\tmp.py", line 13, in <module> print urllib2.urlopen('https://www.google.com').read() File "C:\Python26\lib\urllib2.py", line 124, in urlopen return _opener.open(url, data, timeout)