I have some commands in the [Run] section of my Inno Setup script. Right now, if any of them returns a failure code (non-zero return value), the setup continue
[Run]
You can use the AfterInstall flag in the Run section to trigger the execution of your program and catch the result code.
AfterInstall
Run
See my answer here.
Then according to the result code you can cancel the installation.