mechanize

I'm writing a script(multi-threaded) to retrieve contents from a website, and the site's not very stable so every now and then there's hanging http request which cannot even be time-outed by socket.setdefaulttimeout() . Since I have no control over that website, the only thing I can do is to improve my codes but I'm running out of ideas right now. Sample codes: socket.setdefaulttimeout(150) MechBrowser = mechanize.Browser() Header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (.NET CLR 3.5.30729)'} Url = "http://example.com"

I have a form with <input type="button" name="submit" /> button and would like to be able to click it. I have tried mech.form.click("submit") but that gives the following error: ControlNotFoundError: no control matching kind 'clickable', id 'submit' mech.submit() also doesn't work since its type is button and not submit. Any ideas? Thanks. clicking a type="button" in a pure html form does nothing. For it to do anything, there must be javascript involved. And mechanize doesn't run javascript . So your options are: Read the javascript yourself and simulate with mechanize what it would be doing

I want to use Mechanize to simulate browsing to a web page with active JavaScript, including DOM Events and AJAX, and so far I've found no way to do that. I looked at some Python client browsers that support JavaScript like Spynner and Zope, and none of them really work for me. Spynner crashes PyQt all the time, and Zope doesn't support JavaScript as it seems. Is there a way to simulate browsing with Python only (no extra processes) like WATIR or libraries that manipulate Firefox or Internet Explorer while supporting Javascript fully as if actually browsing the page? I've played with this new

I am trying to import a module from a particular directory. The problem is that if I use sys.path.append(mod_directory) to append the path and then open the python interpreter, the directory mod_directory gets added to the end of the list sys.path. If I export the PYTHONPATH variable before opening the python interpreter, the directory gets added to the start of the list. In the latter case I can import the module but in the former, I cannot. Can somebody explain why this is happening and give me a solution to add the mod_directory to the start, inside a python script ? Ned Deily This is

I'm looking for a library that has functionality similar to Perl's WWW::Mechanize , but for PHP. Basically, it should allow me to submit HTTP GET and POST requests with a simple syntax, and then parse the resulting page and return in a simple format all forms and their fields, along with all links on the page. I know about CURL, but it's a little too barebones, and the syntax is pretty ugly (tons of curl_foo($curl_handle, ...) statements Clarification: I want something more high-level than the answers so far. For example, in Perl, you could do something like: # navigate to the main page $mech-