OK, it might be a problem with my system but I\'m having a very difficult time getting Windows 7 to recognize my custom exe as the default handler of the mailto protocol.
Though a pretty late answer, what worked out for me is this: On regedit, this supersedes the registry keys mentioned in other answers:
HKEY_CURRENT_USER\Software\Microsoft\Windows\Shell\Associations\URLAssociations\MAILTO\Userchoice
Have a look at this page: Default Mailto Registry