I have created a Windows Installer for a windows service with VS 2017. The windows service has a project installer and service installer class inside it. When I cancel the unins
MSVS Installers & Custom Actions: You are using the Microsoft Visual Studio 2017 Installer Projects? They are very limited. The use of custom actions for service installation is one symptom of this. The use of custom actions in general is a very error prone approach (might be an over the top write-up), and for service installation it is almost never the right approach. Get rid of custom actions whenever you can and your deployment failure rate will drop.
Built-In Service Installation Features: You should use the built-in service installation features of Windows Installer (ServiceInstall and ServiceControl tables - which you seem familiar with), and there are a number of tools that can help you do this. Here are some alternative tools (free and commercial) that you can use. WiX does the job nicely, but does have a learning curve.
Service State: I would reboot in your case, to get rid of the service's current error state. It might be in some odd state pending deletion that could cause weird test results. I never take the time to investigate such issues properly, instead prefering to try and eliminate them by a simple reboot to get rid of the error state. Then do the deployment right with built-in features and that error state should never occur. A problem removed is ten problems solved? ( wishful thinking :-) ).