I have created a Windows Installer for a windows service with VS 2017. The windows service has a project installer and service installer class inside it. When I cancel the unins
First make sure that you've done all the custom action service installer nodes, in particular have the Commit and Rollback node custom actions populated. These custom actions all work together, so if you have missed one of them that could cause the problem. In theory, canceling the uninstall should have noted that the service is no longer installed, and it should be re-installed.
If that's not the issue, then it's probably a bug. The most useful thing you can do to fix it is to override base.Uninstall() in the installer class and add some code to check the service is actually installed before calling base.Uninstall().
Visual Studio setup projects are the only ones that use code to install services, and they are custom actions to call installer class methods. Everyone else uses the ServiceInstall and ServiceControl tables supplied by Windows Installer.
Some of the options you can use if you don't want to completely switch to another tool such as WiX are:
http://www.installsite.org/pages/en/msi/tips.htm