I have a MSM file from a third party tool. We customize the MSM with a small script and this tool is integrated into our installer and works fine. All modification is done w
Transforms & msiexec.exe: With your updated information I would say that you can solve all of this by using a transform, or possibly even by setting a property via the installation command line.
Transforms allow you to change "anything" in an MSI / MSM (they are little database fragments applied at runtime to change the MSI / MSM), setting a PUBLIC PROPERTY (always uppercase) allows you to affect whatever that property is designed to affect. So the latter is a "light weight" customization of the MSI / MSM added and supported by the vendor, that is great when available but in my experience rarely sufficient (typical use is to add a license key or a server IP or URL or install only part of the MSI - a few features - simple stuff like that).
Descriptions of how to use transforms or the msiexec.exe command line to customize an MSI installation can be found here:
GUI: I really don't understand what you refer to when the MSM "installs in its own GUI"? Hopefully this is just a shortcut to launch an executable from the start menu? If so you can easily remove it using a transform.
Create Transform: You can create transforms using Orca (MS SDK tool) or any other MSI Deployment Tool, or you can create it programatically using the MSI API (Win32, COM, .NET).
COM Automation: The Microsoft MSI SDK contains a VBScript called WiGenXfm.vbs. You will have it installed if you got Visual Studio installed. You could also find it on github.com (but get it from your own disk to be sure).
DTF / .NET is a component of WiX which allows you to access the MSI API via managed code. The interop is done for you. Sample here.
C++ / Win32: And down-to-the-metal Win32 (accept no substitute - layers be gone :-)).
Note: DTF is installed as part of the WiX toolkit. The open source toolkit capable of building MSI files (quick start suggestions). After installation find the dll's in the WiX installation folder under %ProgramFiles (x86)%. Sub folder "bin". File descriptions here. And I will end with a quick sample of DTF in action getting some properties from an MSI file.