urllib2

So I have a list of sub-reddits and I'm using urllib to open them. As I go through them eventually urllib fails with: urllib2.HTTPError: HTTP Error 429: Unknown Doing some research I found that reddit limits the ammount of requests to their servers by IP: Make no more than one request every two seconds. There's some allowance for bursts of requests, but keep it sane. In general, keep it to no more than 30 requests in a minute. So I figured I'd use time.sleep() to limit my requests to one page each 10 seconds. This ends up failing just as well. The quote above is grabbed from the reddit API

I want to be able to download a page and all of its associated resources (images, style sheets, script files, etc) using Python. I am (somewhat) familiar with urllib2 and know how to download individual urls, but before I go and start hacking at BeautifulSoup + urllib2 I wanted to be sure that there wasn't already a Python equivalent to "wget --page-requisites http://www.google.com ". Specifically I am interested in gathering statistical information about how long it takes to download an entire web page, including all resources. Thanks Mark Websucker? See http://effbot.org/zone/websucker.htm

I'm new to Python and am playing around with making a very basic web crawler. For instance, I have made a simple function to load a page that shows the high scores for an online game. So I am able to get the source code of the html page, but I need to draw specific numbers from that page. For instance, the webpage looks like this: http://hiscore.runescape.com/hiscorepersonal.ws?user1=bigdrizzle13 where 'bigdrizzle13' is the unique part of the link. The numbers on that page need to be drawn out and returned. Essentially, I want to build a program that all I would have to do is type in

I want to to test my application's handling of timeouts when grabbing data via urllib2, and I want to have some way to force the request to timeout. Short of finding a very very slow internet connection, what method can I use? I seem to remember an interesting application/suite for simulating these sorts of things. Maybe someone knows the link? I usually use netcat to listen on port 80 of my local machine: nc -l 80 Then I use http://localhost/ as the request URL in my application. Netcat will answer at the http port but won't ever give a response, so the request is guaranteed to time out

I'm currently trying to initiate a file upload with urllib2 and the urllib2_file library. Here's my code: import sys import urllib2_file import urllib2 URL='http://aquate.us/upload.php' d = [('uploaded', open(sys.argv[1:]))] req = urllib2.Request(URL, d) u = urllib2.urlopen(req) print u.read() I've placed this .py file in my My Documents directory and placed a shortcut to it in my Send To folder (the shortcut URL is ). When I right click a file, choose Send To, and select Aquate (my python), it opens a command prompt for a split second and then closes it. Nothing gets uploaded. I knew there

import urllib print urllib.urlopen('http://www.reefgeek.com/equipment/Controllers_&_Monitors/Neptune_Systems_AquaController/Apex_Controller_&_Accessories/').read() The above script works and returns the expected results while: import urllib2 print urllib2.urlopen('http://www.reefgeek.com/equipment/Controllers_&_Monitors/Neptune_Systems_AquaController/Apex_Controller_&_Accessories/').read() throws the following error: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.5/urllib2.py", line 124, in urlopen return _opener.open(url, data) File "/usr/lib

The urllib2 documentation says that timeout parameter was added in Python 2.6. Unfortunately my code base has been running on Python 2.5 and 2.4 platforms. Is there any alternate way to simulate the timeout? All I want to do is allow the code to talk the remote server for a fixed amount of time. Perhaps any alternative built-in library? (Don't want install 3rd party, like pycurl) you can set a global timeout for all socket operations (including HTTP requests) by using: socket.setdefaulttimeout() like this: import urllib2 import socket socket.setdefaulttimeout(30) f = urllib2.urlopen('http:/