web-crawler

I am looking at writing my own, but I am wondering if there are any good web crawlers out there which are written in Ruby. Short of a full-blown web crawler, any gems that might be helpful in building a web crawler would be useful. I know this part of the question is touched upon in a couple of places, but a list of gems applicable to building a web crawler would be a great resource as well. Felipe Lima I am building wombat, a Ruby DSL to crawl web pages and extract content. Check it out on github https://github.com/felipecsl/wombat It is still in an early stage but is already functional with

I have come across an interview question "If you were designing a web crawler, how would you avoid getting into infinite loops? " and I am trying to answer it. How does it all begin from the beginning. Say Google started with some hub pages say hundreds of them (How these hub pages were found in the first place is a different sub-question). As Google follows links from a page and so on, does it keep making a hash table to make sure that it doesn't follow the earlier visited pages. What if the same page has 2 names (URLs) say in these days when we have URL shorteners etc.. I have taken Google

I have the crawler implemented as below. It is working and it would go through sites regulated under the link extractor . Basically what I am trying to do is to extract information from different places in the page: - href and text() under the class 'news' ( if exists) - image url under the class 'think block' ( if exists) I have three problems for my scrapy: 1) duplicating linkextractor It seems that it will duplicate processed page. ( I check against the export file and found that the same ~.img appeared many times while it is hardly possible) And the fact is , for every page in the website,

This website lists over 250 courses in one list. I want to get the name of each course and insert that into my mysql database using php. The courses are listed like this: <td> computer science</td> <td> media studeies</td> … Is there a way to do that in PHP, instead of me having a mad data entry nightmare? Regular expressions work well. $page = // get the page $page = preg_split("/\n/", $page); for ($text in $page) { $matches = array(); preg_match("/^<td>(.*)<\/td>$/", $text, $matches); // insert $matches[1] into the database } See the documentation for preg_match. You can use this HTML

I'm trying write a crawler to download some information, similar to this Stack Overflow post. The answer is useful for creating the filled-in form, but I'm struggling to find a way to submit the form when a submit button is not part of the form. Here is an example: session <- html_session("www.chase.com") form <- html_form(session)[[3]] filledform <- set_values(form, `user_name` = user_name, `usr_password` = usr_password) session <- submit_form(session, filledform) At this point, I receive this error: Error in names(submits)[[1]] : subscript out of bounds How can I make this form submit? Here

My question is really how to do the same thing as a previous question, but in Scrapy 0.14. Using one Scrapy spider for several websites Basically, I have GUI that takes parameters like domain, keywords, tag names, etc. and I want to create a generic spider to crawl those domains for those keywords in those tags. I've read conflicting things, using older versions of scrapy, by either overriding the spider manager class or by dynamically creating a spider. Which method is preferred and how do I implement and invoke the proper solution? Thanks in advance. Here is the code that I want to make