This applies to both 2008 and 2010 versions (and probably earlier ones too). Also to both C++ and C# projects.
An initial build (after reboot) runs at normal speed
My fix for very sluggish Visual Studio (building anything took about 1.5-2 min) was to turn off the wireless network.
I had enabled the Wireless Network in addition to the wired one. It seems like my computer tried to use the wireless connection to connect to our local server that is not available to wireless network and that caused the long delay.