I have a very simple installer - copy a single dll to a Program Files subfolder and register it with regsvr32.exe. Works great, but if an older version of the dll is install
If you want a major uprade, start with the WiX MajorUpgrade element. The general rules for an upgrade are:
In your original case, failure to follow rules 1 and 2 meant that Windows thought the same product was already installed and went into Repair mode. That should have been your first warning because a major upgrade looks like a fresh install, not a repair. If you have two entries in Programs&Features it means that one or more of those 4 requirements has not been met. If you get "Another version of this product is already installed" it means you didn't follow rule 1, and variations in behavior are about ProductCode and PackageCode values compared with the installed product.