Why no text colors after using CreateFile(“CONOUT$” …) to redirect the console back to the active screen buffer?

Question

I had an old app based off of this setup (but more advanced): https://stackoverflow.com/a/33441726/1236397

Basically, it runs, processes stuff, and closes, and only

Answer 1

As it turns out, after much digging, it seems there's a mode that does not get set when the application runs in "Windows Application" mode:

ENABLE_VIRTUAL_TERMINAL_INPUT (doc)

This is how to set it:

if (GetConsoleMode(outFile, out var cMode))
    SetConsoleMode(outFile, cMode | ENABLE_VIRTUAL_TERMINAL_INPUT);

However, it still would not work, and this is where my issue was. That if statement will ALWAYS fail because only GENERIC_WRITE was specified for CreateFileW()! Turns out GENERIC_READ ALSO needs to be set, otherwise, you cannot read the current mode. The CreateFileW line should be this:

var outFile = CreateFileW("CONOUT$", GENERIC_WRITE | GENERIC_READ, FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, /*FILE_ATTRIBUTE_NORMAL*/0, IntPtr.Zero);

My final code is different from this, but I wrapped up an example here for others if needed:

var outFile = CreateFileW("CONOUT$", GENERIC_WRITE | GENERIC_READ, FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, /*FILE_ATTRIBUTE_NORMAL*/0, IntPtr.Zero);
var safeHandle = new SafeFileHandle(outFile, true);
SetStdHandle(STD_OUTPUT_HANDLE, outFile);
var fs = new FileStream(safeHandle, FileAccess.Write);
var writer = new StreamWriter(fs) { AutoFlush = true };
Console.SetOut(writer);
if (GetConsoleMode(outFile, out var cMode))
    SetConsoleMode(outFile, cMode | ENABLE_VIRTUAL_TERMINAL_INPUT);

A lot of time was wasted trying to figure this out from info scattered everywhere, so I hope this helps save someone else time. ;)