As an alternative to ManualResetEvent and friends, Windows now provides native support for condition variables. I haven't benchmarked it myself, but there's a good chance your performance will improve when leveraging the native API.
Here's a Code Project article that explains how to access this (relatively new) construct from C#:
A .NET Wrapper for the Vista/Server 2008 Condition Variable