mechanize

I am writing a function for some existing python code that will be passed a Mechanize browser object as a parameter. I fill in some details in a form in the browser, and use response = browser.submit() to move the browser to a new page, and collect some information from it. Unfortunately, I occasionally get the following error: httperror_seek_wrapper: HTTP Error 500: Internal Server Error I've navigated to the page in my own browser, and sure enough, I occasionally see this error directly, so I think this is a server problem, not anything to do with robots.txt , headers or similar. The problem

I have this code require 'mechanize' @agent = Mechanize.new page = @agent.get('http://something.com/?page=1') next_page = page.link_with(:href=>/^?page=2/).click As you can see this code should go to the next page. The next_page should have url http://something.com/?page=2 How to get current url for next_page ? sunnyrjuneja next_page.uri.to_s See http://www.rubydoc.info/gems/mechanize/Mechanize/Page/Link#uri-instance_method and http://ruby-doc.org/stdlib-2.4.1/libdoc/uri/rdoc/URI.html For testing purposes, I did the following in irb: require 'mechanize' @agent = Mechanize.new page = @agent.get

I'm trying to use python and mechanize to send sms from my mobile provider website. The problem is that form has a captcha image. Using mechanize I can get the link to the image, but it's different all the time I access that link. Is there any way to get exact picture from mechanize? This is a rough example of how to get the image, note that mechanize uses cookies so any cookies received will be sent to the server with the request for the image (this is probably what you want ). br = mechanize.Browser() response = br.open('http://example.com') soup = BeautifulSoup(response.get_data()) img =

I have page with js that post data via XMLHttpRequest and server side script check for this header, how to send this header? agent = WWW::Mechanize.new { |a| a.user_agent_alias = 'Mac Safari' a.log = Logger.new('./site.log') } agent.post('http://site.com/board.php', { 'act' => '_get_page', "gid" => 1, 'order' => 0, 'page' => 2 } ) do |page| p page end avout I found this post with a web search (two months later, I know) and just wanted to share another solution. You can add custom headers without monkey patching Mechanize using a pre-connect hook: agent = WWW::Mechanize.new agent.pre_connect

The site I'm trying to spider is using the javascript: request.open("POST", url, true); To pull in extra information over ajax that I need to spider. I've tried various permutations of: r = mechanize.urlopen("https://site.tld/dir/" + url, urllib.urlencode({'none' : 'none'})) to get Mechanize to get the page but it always results in me getting the login HTML again, indicating that something is wrong. Firefox doesn't seem to add any HTTP data to the POST according to Firebug, and I'm adding an empty field to try and force the urlopen to use "POST" instead of "GET" hoping the site ignores the

I'd like to know how to select options in a form that is formatted like <td align="left"> <select name="FORM1" id="FORM1" multiple="multiple" size="5"> <option value="Value1">Value1</option> <option value="Value2">Value2</option> </select> </td> Right now, I am using mechanize to connect to the website and traverse to the desired page. This page has many forms such as FORM1, FORM2, FORM3, etc. with options. I'd like to select (enable) Value1 then tell the instance of mechanize to hit the submit button. Which would be a quick way to enable an option based on the form name? senderle Here are

Closed. This question is off-topic. It is not currently accepting answers. Learn more . Want to improve this question? Update the question so it's on-topic for Stack Overflow. I'm using Python 3.6 while I have to fill in a form. Unfortunately, mechanize doesn't work on Python 3. What do you suggest as an alternative to mechanize? SeleniumRC with selenium.py is an alternative (and one of the few workable options if the pages you need to scrape have an important, "structural" role for Javascript operations, esp. AJAX-y ones, since Mechanize doesn't execute the Javascript on the pages it's