The simplest way to do this is to use an external service like an API to block VPN or proxy users.
MaxMind and GetIPIntel both offer it via API, you might want to give it a try. GetIPIntel provides free API service so I suggest you try that first.
For OpenVPN, someone used unique MSS values to identify VPN connections but the setup is complicated and it might be "patched" now.
The strategies you've mentioned in your edits don't seem like a very good idea because you'll run into many false positives. Sending out port scans whenever they connect to your service is going to take a lot of time and resources before you get the results.
