I once wrote a Crawler in .NET. In order to improve its scalability, I tried to take advantage of asynchronous API of .NET.
The System.Net.HttpWebRequest has asynch
This is happening when you use async Send (BeginSend) method of a socket. If you use your own custom threadpool, and send the data over thread with synched Send method is mostly solving this problem. Tested and proved.