screen-scraping

I need to scrape data from a site, but it requires my login first. I've been using hpricot to successfully scrape other sites, but I'm new to using mechanize, and I'm truly baffled by how to work it. I see this example commonly quoted: require 'rubygems' require 'mechanize' a = Mechanize.new a.get('http://rubyforge.org/') do |page| # Click the login link login_page = a.click(page.link_with(:text => /Log In/)) # Submit the login form my_page = login_page.form_with(:action => '/account/login.php') do |f| f.form_loginname = ARGV[0] f.form_pw = ARGV[1] end.click_button my_page.links.each do |link|

I seek a tool that can be run on the command line like so: tablescrape 'http://someURL.foo.com' [n] If n is not specified and there's more than one HTML table on the page, it should summarize them (header row, total number of rows) in a numbered list. If n is specified or if there's only one table, it should parse the table and spit it to stdout as CSV or TSV. Potential additional features: To be really fancy you could parse a table within a table, but for my purposes -- fetching data from wikipedia pages and the like -- that's overkill. An option to asciify any unicode. An option to apply an

I would like to create a program that will enter a string into the text box on a site like Google (without using their public API) and then submit the form and grab the results. Is this possible? Grabbing the results will require the use of HTML scraping I would assume, but how would I enter data into the text field and submit the form? Would I be forced to use a public API? Is something like this just not feasible? Would I have to figure out query strings/parameters? Thanks Theory What I would do is create a little program that can automatically submit any form data to any place and come back

Using BeautifulSoup, I am aiming to scrape the text associated with this HTML hook: <p class="review_comment"> So, using the simple code as follows, content = page.read() soup = BeautifulSoup(content) results = soup.find_all("p", "review_comment") I am happily parsing the text that is living here: <p class="review_comment"> This place is terrible!</p> The bad news is that every 30 or so times the soup.find_all gets a match, it also matches and grabs something that I really don't want, which is a user's old review that they've since updated: <p class="review_comment"> It's 1999, and I will

I'm interested in public facing sites (nothing behind a login / authentication) that have things like: High use of internal 301 and 302 redirects Anti-scraping measures (but not banning crawlers via robots.txt) Non-semantic, or invalid mark-up Content loaded via AJAX in the form of onclicks or infinite scrolling Lots of parameters used in urls Canonical problems Convoluted internal link structure and anything else that generally makes crawling a website a headache! I have built a crawler / spider that performs a range of analysis on a website, and I'm on the lookout for sites that will make it