How to debug a COM C++ Out of Process Server ? (client: Excel vba)

Question

I don't know how to debug a COM ATL C++ out of process exe server with Visual Studio 2015.

For a COM ATL DLL there is no problem, the breakpoints are automatically triggered when I call the COM dll. However when I try to debug an EXE server, the debugger didn't break. Indeed the exe server is only loaded at demand and the debug symbols of the exe server are not loaded when the client starts.

My client, that call the COM server, is Excel VBA (I have added the reference to the com exe, in the VBA editor, via Tools/References). To debug it, I set the path to Excel.exe in the Visual Studio Debug command and as argument I set the path to my xlsm file. Then I just call call the VBA code in Excel. It works for the com DLL but not for the COM Exe. I also tried by attaching VS to Excel and it didn't work.

I have found this post ( Windows COM - how to debug a COM server ), it seems this is caused by a elevation/permission issue, but I tried the proposed answer but it didn't works.

Does any one can help me with this ?

How do you debug an out-of-process com server ?

Is it necessary to attach a debugger to the client that consumes the EXE server (so in my case to excel) ?

Answer 1

It's tricky trying to debug an EXE server.

What I would do is modify the registry so that you are prompted to debug the EXE when it is loaded.

Here is the contents of an exported .reg file used to modify the registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\YourExeName.exe]
"debugger"="vsjitdebugger.exe"

Change YourExeName.exe to something more useful...

When your EXE server is launched, you will be prompted to launch a debugger or attach from a current running instance of a debugger. It is faster and better to already have a debugger running... When I say "debugger", a running instance of Visual Studio is fine.

A problem when trying to debug an EXE server is that when you set breakpoints and step through the code, you can exceed the COM timeout and then the EXE client calling your process will have a failed call due to timeout and it will just continue executing. When everything is in process, this is not such a problem. For an EXE, it might be helpful to have a lot more TRACE statements in your debug build in the area where you think you might have problems.

It is not strictly necessary to attach to the client process. Sometimes it is helpful, and sometimes not. It's obviously much more easier if you have the source code to the calling client process.