Effects of switching between /SUBSYSTEM:CONSOLE to /SUBSYSTEM:WINDOWS in a DLL

Question

I couldn\'t find anything useful on the MSDN for this case. Using Dependency Walker, in the module list, I see a mixed usage of Console and GUI.

Does this have an im

Answer 1

My below reply is just some findings.

I tried to create below types of project within VS2015:

• Win32 console app project
• Win32 DLL project
• Win32 Windows app project

Below are their complete linking options:

Win32 console app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\ConsoleApplication3.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 DLL project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.dll" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.lib" /DEBUG /DLL /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32DLLProject1.dll.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 Windows app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32WindowsProject1.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

So we can see, for a /DLL, /SUBSYSTEM:WINDOWS is defined.

Then I tried to build a DLL with 3 different SUBSYSTEM values:

• CONSOLE
• WINDOWS
• NOT SET

There are other values as well, but I can only build successfully with the above 3. Other values will lead to linking errors, which means some external symbols are needed.

The full list of possible SUBSYSTEM values is:

You can use the CFF Explorer to inspect the PE/COFF binaries.

The SUBSYSTEM header field locates at the file offset 0x14C of a PE/COFF file. It is part of the Optional Header.

/DLL /SUBSYESTEM:CONSOLE

/DLL /SUBSYESTEM:WINDOWS

/DLL and no /SUBSYESTEM option (by selecting NOT SET in VS2015 project properties page

Interestingly, the NOT SET and WINDOWS values lead to the same content in the binary header.

And I compared the whole binary of the 3 DLLs. It seems the rest of the binaries are all the same, except for some timestamp and debug info.

This is just some fact I found. How /SUBSYSTEM option affects the binary behavior depends on how the loader interprets this field.

In one of the projects I participated in, the /SUBSYSTEM:CONSOLE and /DLL are used together, which is different from the default combination for a DLL project. But it seems nothing bad happens. So I agree with @Frédéric Hamidi that this flag has no functional impact to a DLL.