agents

Below is a sample robots.txt file to Allow multiple user agents with multiple crawl delays for each user agent. The Crawl-delay values are for illustration purposes and will be different in a real robots.txt file. I have searched all over the web for proper answers but could not find one. There are too many mixed suggestions and I do not know which is the correct / proper method. Questions: (1) Can each user agent have it's own crawl-delay? (I assume yes) (2) Where do you put the crawl-delay line for each user agent, before or after the Allow / Dissallow line? (3) Does there have to be a blank

After playing with F# agents I tried to do a map reduce using them. The basic structure I use is: map supervisor which queues up all the work to do in its state and receives work request from map workers reduce supervisor does the same thing as map supervisor for reduce work a bunch of map and reduce workers that map and reduce, if one fails its work it sends it back to the respective supervisr to be reprocessed. The questions I wonder about is: does this make any sense compared to a more traditional (yet very nice) map reduce like (http://tomasp.net/blog/fsharp-parallel-aggregate.aspx) that