.NET Does NOT Have Reliable Asynchronouos Socket Communication?

Question

I once wrote a Crawler in .NET. In order to improve its scalability, I tried to take advantage of asynchronous API of .NET.

The System.Net.HttpWebRequest has asynch

Answer 1

Hmya, this is not a .NET framework problem. The linked KB article could have been a bit more explicit: "you're using a loaded gun, this is what happens when you aim it at your foot". The bullets in that gun are .NET giving you the ability to start as many asynchronous I/O requests as you dare. It will do what you ask it to do, until you hit some kind of resource limit. In this case, probably, having too many pinned receive buffers in the generation 0 heap.

Resource management is still very much our job, not .NET's. It is no different from allocating memory without bound. Solving this particular problem requires you to put a limit on the number of uncompleted BeginGetResponse() requests. Having hundreds of them make little sense, every one of them has to squeeze through the Intertube one at a time. Adding another request will just cause it to take longer to complete. Or crash your program.