Designing a web crawler
I have come across an interview question \"If you were designing a web crawler, how would you avoid getting into infinite loops? \" and I am trying to answer it.
How
-
You'd have to have some sort of hash table to store the results in, you'd just have to check it before each page load.
讨论(0) -
The web crawler is a computer program which used to collect/crawling following key values(HREF links, Image links, Meta Data .etc) from given website URL. It is designed like intelligent to follow different HREF links which are already fetched from the previous URL, so in this way, Crawler can jump from one website to other websites. Usually, it called as a Web spider or Web Bot. This mechanism always acts as a backbone of the Web search engine.
Please find the source code from my tech blog - http://www.algonuts.info/how-to-built-a-simple-web-crawler-in-php.html
<?php class webCrawler { public $siteURL; public $error; function __construct() { $this->siteURL = ""; $this->error = ""; } function parser() { global $hrefTag,$hrefTagCountStart,$hrefTagCountFinal,$hrefTagLengthStart,$hrefTagLengthFinal,$hrefTagPointer; global $imgTag,$imgTagCountStart,$imgTagCountFinal,$imgTagLengthStart,$imgTagLengthFinal,$imgTagPointer; global $Url_Extensions,$Document_Extensions,$Image_Extensions,$crawlOptions; $dotCount = 0; $slashCount = 0; $singleSlashCount = 0; $doubleSlashCount = 0; $parentDirectoryCount = 0; $linkBuffer = array(); if(($url = trim($this->siteURL)) != "") { $crawlURL = rtrim($url,"/"); if(($directoryURL = dirname($crawlURL)) == "http:") { $directoryURL = $crawlURL; } $urlParser = preg_split("/\//",$crawlURL); //-- Curl Start -- $curlObject = curl_init($crawlURL); curl_setopt_array($curlObject,$crawlOptions); $webPageContent = curl_exec($curlObject); $errorNumber = curl_errno($curlObject); curl_close($curlObject); //-- Curl End -- if($errorNumber == 0) { $webPageCounter = 0; $webPageLength = strlen($webPageContent); while($webPageCounter < $webPageLength) { $character = $webPageContent[$webPageCounter]; if($character == "") { $webPageCounter++; continue; } $character = strtolower($character); //-- Href Filter Start -- if($hrefTagPointer[$hrefTagLengthStart] == $character) { $hrefTagLengthStart++; if($hrefTagLengthStart == $hrefTagLengthFinal) { $hrefTagCountStart++; if($hrefTagCountStart == $hrefTagCountFinal) { if($hrefURL != "") { if($parentDirectoryCount >= 1 || $singleSlashCount >= 1 || $doubleSlashCount >= 1) { if($doubleSlashCount >= 1) { $hrefURL = "http://".$hrefURL; } else if($parentDirectoryCount >= 1) { $tempData = 0; $tempString = ""; $tempTotal = count($urlParser) - $parentDirectoryCount; while($tempData < $tempTotal) { $tempString .= $urlParser[$tempData]."/"; $tempData++; } $hrefURL = $tempString."".$hrefURL; } else if($singleSlashCount >= 1) { $hrefURL = $urlParser[0]."/".$urlParser[1]."/".$urlParser[2]."/".$hrefURL; } } $host = ""; $hrefURL = urldecode($hrefURL); $hrefURL = rtrim($hrefURL,"/"); if(filter_var($hrefURL,FILTER_VALIDATE_URL) == true) { $dump = parse_url($hrefURL); if(isset($dump["host"])) { $host = trim(strtolower($dump["host"])); } } else { $hrefURL = $directoryURL."/".$hrefURL; if(filter_var($hrefURL,FILTER_VALIDATE_URL) == true) { $dump = parse_url($hrefURL); if(isset($dump["host"])) { $host = trim(strtolower($dump["host"])); } } } if($host != "") { $extension = pathinfo($hrefURL,PATHINFO_EXTENSION); if($extension != "") { $tempBuffer =""; $extensionlength = strlen($extension); for($tempData = 0; $tempData < $extensionlength; $tempData++) { if($extension[$tempData] != "?") { $tempBuffer = $tempBuffer.$extension[$tempData]; continue; } else { $extension = trim($tempBuffer); break; } } if(in_array($extension,$Url_Extensions)) { $type = "domain"; } else if(in_array($extension,$Image_Extensions)) { $type = "image"; } else if(in_array($extension,$Document_Extensions)) { $type = "document"; } else { $type = "unknown"; } } else { $type = "domain"; } if($hrefURL != "") { if($type == "domain" && !in_array($hrefURL,$this->linkBuffer["domain"])) { $this->linkBuffer["domain"][] = $hrefURL; } if($type == "image" && !in_array($hrefURL,$this->linkBuffer["image"])) { $this->linkBuffer["image"][] = $hrefURL; } if($type == "document" && !in_array($hrefURL,$this->linkBuffer["document"])) { $this->linkBuffer["document"][] = $hrefURL; } if($type == "unknown" && !in_array($hrefURL,$this->linkBuffer["unknown"])) { $this->linkBuffer["unknown"][] = $hrefURL; } } } } $hrefTagCountStart = 0; } if($hrefTagCountStart == 3) { $hrefURL = ""; $dotCount = 0; $slashCount = 0; $singleSlashCount = 0; $doubleSlashCount = 0; $parentDirectoryCount = 0; $webPageCounter++; while($webPageCounter < $webPageLength) { $character = $webPageContent[$webPageCounter]; if($character == "") { $webPageCounter++; continue; } if($character == "\"" || $character == "'") { $webPageCounter++; while($webPageCounter < $webPageLength) { $character = $webPageContent[$webPageCounter]; if($character == "") { $webPageCounter++; continue; } if($character == "\"" || $character == "'" || $character == "#") { $webPageCounter--; break; } else if($hrefURL != "") { $hrefURL .= $character; } else if($character == "." || $character == "/") { if($character == ".") { $dotCount++; $slashCount = 0; } else if($character == "/") { $slashCount++; if($dotCount == 2 && $slashCount == 1) $parentDirectoryCount++; else if($dotCount == 0 && $slashCount == 1) $singleSlashCount++; else if($dotCount == 0 && $slashCount == 2) $doubleSlashCount++; $dotCount = 0; } } else { $hrefURL .= $character; } $webPageCounter++; } break; } $webPageCounter++; } } $hrefTagLengthStart = 0; $hrefTagLengthFinal = strlen($hrefTag[$hrefTagCountStart]); $hrefTagPointer =& $hrefTag[$hrefTagCountStart]; } } else { $hrefTagLengthStart = 0; } //-- Href Filter End -- //-- Image Filter Start -- if($imgTagPointer[$imgTagLengthStart] == $character) { $imgTagLengthStart++; if($imgTagLengthStart == $imgTagLengthFinal) { $imgTagCountStart++; if($imgTagCountStart == $imgTagCountFinal) { if($imgURL != "") { if($parentDirectoryCount >= 1 || $singleSlashCount >= 1 || $doubleSlashCount >= 1) { if($doubleSlashCount >= 1) { $imgURL = "http://".$imgURL; } else if($parentDirectoryCount >= 1) { $tempData = 0; $tempString = ""; $tempTotal = count($urlParser) - $parentDirectoryCount; while($tempData < $tempTotal) { $tempString .= $urlParser[$tempData]."/"; $tempData++; } $imgURL = $tempString."".$imgURL; } else if($singleSlashCount >= 1) { $imgURL = $urlParser[0]."/".$urlParser[1]."/".$urlParser[2]."/".$imgURL; } } $host = ""; $imgURL = urldecode($imgURL); $imgURL = rtrim($imgURL,"/"); if(filter_var($imgURL,FILTER_VALIDATE_URL) == true) { $dump = parse_url($imgURL); $host = trim(strtolower($dump["host"])); } else { $imgURL = $directoryURL."/".$imgURL; if(filter_var($imgURL,FILTER_VALIDATE_URL) == true) { $dump = parse_url($imgURL); $host = trim(strtolower($dump["host"])); } } if($host != "") { $extension = pathinfo($imgURL,PATHINFO_EXTENSION); if($extension != "") { $tempBuffer =""; $extensionlength = strlen($extension); for($tempData = 0; $tempData < $extensionlength; $tempData++) { if($extension[$tempData] != "?") { $tempBuffer = $tempBuffer.$extension[$tempData]; continue; } else { $extension = trim($tempBuffer); break; } } if(in_array($extension,$Url_Extensions)) { $type = "domain"; } else if(in_array($extension,$Image_Extensions)) { $type = "image"; } else if(in_array($extension,$Document_Extensions)) { $type = "document"; } else { $type = "unknown"; } } else { $type = "domain"; } if($imgURL != "") { if($type == "domain" && !in_array($imgURL,$this->linkBuffer["domain"])) { $this->linkBuffer["domain"][] = $imgURL; } if($type == "image" && !in_array($imgURL,$this->linkBuffer["image"])) { $this->linkBuffer["image"][] = $imgURL; } if($type == "document" && !in_array($imgURL,$this->linkBuffer["document"])) { $this->linkBuffer["document"][] = $imgURL; } if($type == "unknown" && !in_array($imgURL,$this->linkBuffer["unknown"])) { $this->linkBuffer["unknown"][] = $imgURL; } } } } $imgTagCountStart = 0; } if($imgTagCountStart == 3) { $imgURL = ""; $dotCount = 0; $slashCount = 0; $singleSlashCount = 0; $doubleSlashCount = 0; $parentDirectoryCount = 0; $webPageCounter++; while($webPageCounter < $webPageLength) { $character = $webPageContent[$webPageCounter]; if($character == "") { $webPageCounter++; continue; } if($character == "\"" || $character == "'") { $webPageCounter++; while($webPageCounter < $webPageLength) { $character = $webPageContent[$webPageCounter]; if($character == "") { $webPageCounter++; continue; } if($character == "\"" || $character == "'" || $character == "#") { $webPageCounter--; break; } else if($imgURL != "") { $imgURL .= $character; } else if($character == "." || $character == "/") { if($character == ".") { $dotCount++; $slashCount = 0; } else if($character == "/") { $slashCount++; if($dotCount == 2 && $slashCount == 1) $parentDirectoryCount++; else if($dotCount == 0 && $slashCount == 1) $singleSlashCount++; else if($dotCount == 0 && $slashCount == 2) $doubleSlashCount++; $dotCount = 0; } } else { $imgURL .= $character; } $webPageCounter++; } break; } $webPageCounter++; } } $imgTagLengthStart = 0; $imgTagLengthFinal = strlen($imgTag[$imgTagCountStart]); $imgTagPointer =& $imgTag[$imgTagCountStart]; } } else { $imgTagLengthStart = 0; } //-- Image Filter End -- $webPageCounter++; } } else { $this->error = "Unable to proceed, permission denied"; } } else { $this->error = "Please enter url"; } if($this->error != "") { $this->linkBuffer["error"] = $this->error; } return $this->linkBuffer; } } ?>讨论(0) -
If you want to get a detailed answer take a look at section 3.8 this paper, which describes the URL-seen test of a modern scraper:
In the course of extracting links, any Web crawler will encounter multiple links to the same document. To avoid downloading and processing a document multiple times, a URL-seen test must be performed on each extracted link before adding it to the URL frontier. (An alternative design would be to instead perform the URL-seen test when the URL is removed from the frontier, but this approach would result in a much larger frontier.)
To perform the URL-seen test, we store all of the URLs seen by Mercator in canonical form in a large table called the URL set. Again, there are too many entries for them all to fit in memory, so like the document fingerprint set, the URL set is stored mostly on disk.
To save space, we do not store the textual representation of each URL in the URL set, but rather a fixed-sized checksum. Unlike the fingerprints presented to the content-seen test’s document fingerprint set, the stream of URLs tested against the URL set has a non-trivial amount of locality. To reduce the number of operations on the backing disk file, we therefore keep an in-memory cache of popular URLs. The intuition for this cache is that links to some URLs are quite common, so caching the popular ones in memory will lead to a high in-memory hit rate.
In fact, using an in-memory cache of 2^18 entries and the LRU-like clock replacement policy, we achieve an overall hit rate on the in-memory cache of 66.2%, and a hit rate of 9.5% on the table of recently-added URLs, for a net hit rate of 75.7%. Moreover, of the 24.3% of requests that miss in both the cache of popular URLs and the table of recently-added URLs, about 1=3 produce hits on the buffer in our random access file implementation, which also resides in user-space. The net result of all this buffering is that each membership test we perform on the URL set results in an average of 0.16 seek and 0.17 read kernel calls (some fraction of which are served out of the kernel’s file system buffers). So, each URL set membership test induces one-sixth as many kernel calls as a membership test on the document fingerprint set. These savings are purely due to the amount of URL locality (i.e., repetition of popular URLs) inherent in the stream of URLs encountered during a crawl.
Basically they hash all of the URLs with a hashing function that guarantees unique hashes for each URL and due to the locality of URLs, it becomes very easy to find URLs. Google even open-sourced their hashing function: CityHash
WARNING!
They might also be talking about bot traps!!! A bot trap is a section of a page that keeps generating new links with unique URLs and you will essentially get trapped in an "infinite loop" by following the links that are being served by that page. This is not exactly a loop, because a loop would be the result of visiting the same URL, but it's an infinite chain of URLs which you should avoid crawling.Update 12/13/2012
- the day after the world was supposed to end :)Per Fr0zenFyr's comment: if one uses the AOPIC algorithm for selecting pages, then it's fairly easy to avoid bot-traps of the infinite loop kind. Here is a summary of how AOPIC works:
- Get a set of N seed pages.
- Allocate X amount of credit to each page, such that each page has X/N credit (i.e. equal amount of credit) before crawling has started.
- Select a page P, where the P has the highest amount of credit (or if all pages have the same amount of credit, then crawl a random page).
- Crawl page P (let's say that P had 100 credits when it was crawled).
- Extract all the links from page P (let's say there are 10 of them).
- Set the credits of P to 0.
- Take a 10% "tax" and allocate it to a Lambda page.
- Allocate an equal amount of credits each link found on page P from P's original credit - the tax: so (100 (P credits) - 10 (10% tax))/10 (links) = 9 credits per each link.
- Repeat from step 3.
Since the Lambda page continuously collects tax, eventually it will be the page with the largest amount of credit and we'll have to "crawl" it. I say "crawl" in quotes, because we don't actually make an HTTP request for the Lambda page, we just take its credits and distribute them equally to all of the pages in our database.
Since bot traps only give internal links credits and they rarely get credit from the outside, they will continually leak credits (from taxation) to the Lambda page. The Lambda page will distribute that credits out to all of the pages in the database evenly and upon each cycle the bot trap page will lose more and more credits, until it has so little credits that it almost never gets crawled again. This will not happen with good pages, because they often get credits from back-links found on other pages. This also results in a dynamic page rank and what you will notice is that any time you take a snapshot of your database, order the pages by the amount of credits they have, then they will most likely be ordered roughly according to their true page rank.
This only avoid bot traps of the infinite-loop kind, but there are many other bot traps which you should watch out for and there are ways to get around them too.
讨论(0) -
The crawler keeps a URL pool that contains all the URLs to be crawled. To avoid “infinite loop”, the basic idea is to check the existence of each URL before adding to the pool.
However, this is not easy to implement when the system has scaled to certain level. The naive approach is keeping all the URLs in a hashset and check existence of each new URL. This won't work when there are too many URLs to fit into memory.
There are a couple of solutions here. For instance, instead of storing all the URLs into memory, we should keep them in disk. To save space, URL hash should be used instead of the raw URL. It's also worth to note that we should keep the canonical form of URL rather than the original one. So if the URL is shortened by services like bit.ly, it's better to get the final URL. To speed up the checking process, a cache layer can be built. Or you can see it as a distributed cache system, which is a separate topic.
The post Build a Web Crawler has a detailed analysis of this problem.
讨论(0) -
I have also required to use crawler and cant find proper one for my requirement so after that I have developed basic crawler library in order to implement simple requirements. But enabling to fulfill almost all principles of crawler. You can check DotnetCrawler github repo which implement Downloader-Processor-Pipeline modules in its own with default implementation using Entity Framework Core in order to save data into Sql Server.
https://github.com/mehmetozkaya/DotnetCrawler
讨论(0) -
While everybody here already suggested how to create your web crawler, here is how how Google ranks pages.
Google gives each page a rank based on the number of callback links (how many links on other websites point to a specific website/page). This is called relevance score. This is based on the fact that if a page has many other pages link to it, it's probably an important page.
Each site/page is viewed as a node in a graph. Links to other pages are directed edges. A degree of a vertex is defined as the number of incoming edges. Nodes with a higher number of incoming edges are ranked higher.
Here's how the PageRank is determined. Suppose that page Pj has Lj links. If one of those links is to page Pi, then Pj will pass on 1/Lj of its importance to Pi. The importance ranking of Pi is then the sum of all the contributions made by pages linking to it. So if we denote the set of pages linking to Pi by Bi, then we have this formula:
Importance(Pi)= sum( Importance(Pj)/Lj ) for all links from Pi to BiThe ranks are placed in a matrix called hyperlink matrix: H[i,j]
A row in this matrix is either 0, or 1/Lj if there is a link from Pi to Bi. Another property of this matrix is that if we sum all rows in a column we get 1.
Now we need multiply this matrix by an Eigen vector, named I (with eigen value 1) such that:
I = H*INow we start iterating: IH, IIH, IIIH .... I^k *H until the solution converges. ie we get pretty much the same numbers in the matrix in step k and k+1.
Now whatever is left in the I vector is the importance of each page.
For a simple class homework example see http://www.math.cornell.edu/~mec/Winter2009/RalucaRemus/Lecture3/lecture3.html
As for solving the duplicate issue in your interview question, do a checksum on the entire page and use either that or a bash of the checksum as your key in a map to keep track of visited pages.
讨论(0)
加载中...
- 热议问题