urllib2

I was condering when I use urllib2.urlopen() does it just to header reads or does it actually bring back the entire webpage? IE does the HTML page actually get fetch on the urlopen call or the read() call? handle = urllib2.urlopen(url) html = handle.read() The reason I ask is for this workflow... I have a list of urls (some of them with short url services) I only want to read the webpage if I haven't seen that url before I need to call urlopen() and use geturl() to get the final page that link goes to (after the 302 redirects) so I know if I've crawled it yet or not. I don't want to incur the

I'm trying to retrieve the following URL: http://www.winkworth.co.uk/sale/property/flat-for-sale-in-masefield-court-london-n5/HIH140004 . import urllib2 response = urllib2.urlopen('http://www.winkworth.co.uk/rent/property/terraced-house-to-rent-in-mill-road--/WOT140129') response.read() However I'm getting an empty string. When I try it through the browser or with cURL it works fine. Any ideas what's going on? I got a response when using the requests library but not when using urllib2 , so I experimented with HTTP request headers. As it turns out, the server expects an Accept header; urllib2

I am trying to scrap a webpage to get table values from text data returned from requests response. </thead> <tbody class="stats"></tbody> <tbody class="annotation"></tbody> </table> </div> Actually there is some data present inside tbody classes but `I am unable to access that data using requests. Here is my code server = "http://www.ebi.ac.uk/QuickGO/GProtein" header = {'User-agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'} payloads = {'ac':'Q9BRY0'} response = requests.get(server, params=payloads) print(response.text) #soup = BeautifulSoup

I have searched a lot of similar question on SO, but did not find an exact match to my case. I am trying to download a video using python 2.7 Here is my code for downloading the video import urllib2 from bs4 import BeautifulSoup as bs with open('video.txt','r') as f: last_downloaded_video = f.read() webpage = urllib2.urlopen('http://*.net/watch/**-'+last_downloaded_video) soup = bs(webpage) a = [] for link in soup.find_all('a'): if link.has_attr('data-video-id'): a.append(link) #try just with first data-video-id id = a[0]['data-video-id'] webpage2 = urllib2.urlopen('http://*/video/play/'+id)