web-crawler

This question already has an answer here: JavaScript closure inside loops – simple practical example 43 answers I'm sure my problem is based on a lack of understanding of asynch programming in node.js but here goes. For example: I have a list of links I want to crawl. When each asynch request returns I want to know which URL it is for. But, presumably because of race conditions, each request returns with the URL set to the last value in the list. var links = ['http://google.com', 'http://yahoo.com']; for (link in links) { var url = links[link]; require('request')(url, function() { console.log

I'm deploying a replacement site for a client but they don't want all their old pages to end in 404s. Keeping the old URL structure wasn't possible because it was hideous. So I'm writing a 404 handler that should look for an old page being requested and do a permanent redirect to the new page. Problem is, I need a list of all the old page URLs. I could do this manually, but I'd be interested if there are any apps that would provide me a list of relative (eg: /page/path, not http:/.../page/path) URLs just given the home page. Like a spider but one that doesn't care about the content other than

Does someone know a way to request Google to re-crawl a website? If possible, this shouldn't last months. My site is showing an old title in Google's search results. How can I show it with the correct title and description? kevinmicke There are two options. The first (and better) one is using the Fetch as Google option in Webmaster Tools that Mike Flynn commented about. Here are detailed instructions: Go to: https://www.google.com/webmasters/tools/ and log in If you haven't already, add and verify the site with the "Add a Site" button Click on the site name for the one you want to manage Click

I created a php page that is only accessible by means of token/pass received through $_GET Therefore if you go to the following url you'll get a generic or blank page http://fakepage11.com/secret_page.php However if you used the link with the token it shows you special content http://fakepage11.com/secret_page.php?token=344ee833bde0d8fa008de206606769e4 Of course this is not as safe as a login page, but my only concern is to create a dynamic page that is not indexable and only accessed through the provided link. Are dynamic pages that are dependent of $_GET variables indexed by google and other

I want to make a script that's crawling a website and it should return the locations of all the banners showed on that page. The locations of banners are most of the time from known domains. But banners are not in the HTML as an easy image or swf-file. Most of the times a Javascript is used to show the banner. So if a .swf-file or image-file is loaded from a banner-domain, it should return that url. Is that possible to do? And how could I do that roughly? Best would be if it can also returns the landing page of that ad. How to solve that? You could use selenium to open the pages in a real