.NET Does NOT Have Reliable Asynchronouos Socket Communication?

Question

I once wrote a Crawler in .NET. In order to improve its scalability, I tried to take advantage of asynchronous API of .NET.

The System.Net.HttpWebRequest has asynch

Answer 1

This isn't limited to .Net.

It's a simple fact that each async request (file, network, etc) uses memory and (at some point, for networking requests at least) non paged pool (see here for details of the problems you can get in unmanaged code). The number of outstanding requests is therefore limited by the amount of memory. Pre-Vista there were some seriously low non paged pool limits that would cause you problems well before you ran out of memory, but in a post-vista environment things are much better for non paged pool usage (see here).

It's a little more complex in managed code as, in addition to the issues you get in the unmanaged world, you also have to deal with the fact that the memory buffers you use for async requests are pinned until those requests complete. Sounds like you're having these problems with reads, but it's just as bad, if not worse, for writes (as soon as TCP flow control kicks in on a connection those send completions are going to start taking longer to occur and so those buffers are pinned for longer and longer - see here and here).

The problem isn't that the .Net async stuff is broken, just that the abstraction is such that it makes it all look much easier than it really is. For example, to avoid the pinning issue, allocate all of your buffers in a single, large contiguous block at program start up rather than on demand...

Personally I'd write such a crawler in unmanaged code, but that's just me ;) You will still face many of the issues, but you have a bit more control over them.