I have installed the Visual Studio 2010 Remote Debugger on a Windows Server 2003 (x86) server, and am attempting to connect to it results in the following error:
Here are the steps I took to get remote debugging to work against an ASP.NET app. Not sure if you've done this already, hopefully something might help.
On my machine (call it DEVMACHINE from now on) I shared out the folder that contained the remote debugger (msvsmon.exe). On my machine, it was located at C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\Remote Debugger\x86. I called the share msvsmon
On the server, I opened Windows explorer and navigated to \\DEVMACHINE\msvsmon, and ran msvsmon.exe (This opened the Visual Studio Remote Debugging Monitor)
On DEVMACHINE, I started Visual Studio 2010 and opened the solution that represents the application I'm attempting to debug.
In Visual Studio, clicked Tools > Attach To Process...
Entered the server name in the Qualifier field, then double clicked on the w3p.exe process that was in the list.
I then placed a break point in the location I wanted to start debugging
Couple things to note: The code deployed to the server was a Debug Build, the pdb files were there, along with the binaries. I had full admin rights on the server. No tools were installed on the server, I simply ran the exe that was located on DEVMACHINE. I did not have any firewalls between the DEVMACHINE and the server. And, both DEVMACHINE and the server are on the same domain.
Hope that helps.