That may be to the Linger option. See here for more information: Graceful Shutdown, Linger Options, and Socket Closure
in .NET, you can change it with the Socket.SetSocketOption method.
EDIT: if it's not the Linger option, you should try to enable full socket traces, here is a .config sample:
EDIT: or you can hit what's called the TIME_WAIT, described here: Please explain the TIME_WAIT state which is 120ms. It's generally better to use Close() with the SocketOptionName.ReuseAddress rather than to use Disconnect(true). See comments here as well (in the comments section).