OK, it might be a problem with my system but I\'m having a very difficult time getting Windows 7 to recognize my custom exe as the default handler of the mailto protocol.
The way described in the answer which you referenced in your question seems correct way for Windows 7 as for other operation systems.
I suppose that you use 64-bit version of Windows 7 and your program is 32-bit application. So if your application write in the HKEY_LOCAL_MACHINE\SOFTWARE\Classes\mailto\shell\open\command key it modifies only the key HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Classes\mailto\shell\open\command used for 32-bit applications. You can read more about registry virtualisation here and here.
If it's your case you can call directly native Win32 API RegOpenKeyEx with the KEY_WOW64_64KEY flag which will be ignored on 32-bit operation system.