I have a WIX project that must only be installed on Windows Server 2016 (or newer).
Looking at Microsoft documentation, VersionNT for:
Windows Server 2016 is
In light of this article about the values of VersionNT values for Windows 10 and Server 2016:
https://support.microsoft.com/en-us/help/3202260/versionnt-value-for-windows-10-and-windows-server-2016
I would start with a condition of:
VersionNT=603 and MsiNTProductType > 1
where the latter indicates a server system:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa370329(v=vs.85).aspx
Beyond that, look at the WindowsBuild property to see if it has a useful value for Server 2016 (or a value you can use to exclude Server 2012). You could also look at a custom action that calls GetSystemMetrics:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724385(v=vs.85).aspx
to see if SM_SERVERR2 is set in 2012.
You do not need a condition that includes the Installed property. When you first do the install the Installed value is not set. If you try to install the same MSI again you won't get as far as the launch condition because Windows will see that this particular ProductCode is already installed and go into maintenance mode (repair, uninstall, feature change type stuff). So it's not clear why you think you need it. If you need to be sure that launch conditions apply only on the initial install the add the condition "Not Installed" to the LaunchConditions action in the UI and execute sequences.