SetConsoleMode fails with zero, lasterror = 0

Question

This is not a duplicate! - Well, after reading the comments, maybe it is.

I was looking for a way to italicize text in the console output of a console application, in c#, Visual Studio 2015, Targeting .NET Framework 4.5.2, OS = Windows 7.

The Microsoft Documentation is pretty clear

It's here - and it's so misleading it's wrong. This is an OS problem.

I found the following question with a solution that does what I want by Vladimir Reshetnikov,

adding text decorations to console output

answered Mar 28 at 19:52 in one of the answers, and code like it in git, and elsewhere... my problem is - naturally - it doesn't work for me.

I copied the author's code with minor mods into the following console application

using System;
using System.Runtime.InteropServices;

namespace ConsoleApplication1
{
    class Program
    {
        const int STD_OUTPUT_HANDLE = -11;
        const uint ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4;

        [DllImport("kernel32.dll", SetLastError = true)]
        static extern IntPtr GetStdHandle(int nStdHandle);

        [DllImport("kernel32.dll")]
        static extern bool GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode);

        [DllImport("kernel32.dll")]
        static extern bool SetConsoleMode(IntPtr hConsoleHandle, uint dwMode);

        static void Main()
        {
            var handle = GetStdHandle(STD_OUTPUT_HANDLE);
            uint mode;
            GetConsoleMode(handle, out mode);
            mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
            SetConsoleMode(handle, mode);

            const string UNDERLINE = "\x1B[4m";
            const string RESET = "\x1B[0m";
            Console.WriteLine("Some " + UNDERLINE + "underlined" + RESET + " text");
            Console.ReadLine();
        }
    }
}

and I get the VT commands in the window, instead of underline, as in the article.

Here's my console window:

I've trapped the return value from ConsoleSetMode - it's zero. I've seen this failure with lasterror = 6, but the lasterror here is 0.

Think it's a recent update? ... or something? [edit] It's a Windows version problem - Windows 10 AU, apparently, is required.

Answer 1

Make sure the checkbox "Use legacy console" near the bottom of the console properties is not set:

If you do not see this checkbox, then you are probably using a too old version of Windows.

You can manipulate this checkbox programmatically using the registry key HKCU\Console\ForceV2 as explained in this answer.

Answer 2

Ok. It was a duplicate - sort of. In the answers to the question, referred to by Gusman, SetConsoleMode() and ENABLE_VIRTUAL_TERMINAL_PROCESSING? (that I couldn't find, but should have) - this functionality is only available on Windows 10 (and further... according to Tamás Deme 'tomzorz', only on or after Windows 10 AU), despite Microsoft's claim that it is available on Windows 2000 "and later".

So, the answer is: it doesn't work, and won't except on Windows 10, which is a dead end until Windows 7 has been removed from the planet, and there's no chance it will have to pass QC on Windows 7.

By that time, Console applications will be forbidden by law.

Answer 3

More error checking is needed.

private static readonly IntPtr InvalidHandle = new IntPtr(-1);

handle = GetStdHandle(STD_OUTPUT_HANDLE);
if (handle == InvalidHandle) {
    throw new System.ComponentModel.Win32Exception(Marshal.GetLastWin32Error());
}
if (!GetConsoleMode(handle, out uint mode)) {
    throw new System.ComponentModel.Win32Exception(Marshal.GetLastWin32Error());
}
mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
if (!SetConsoleMode(handle, mode)) {
    throw new System.ComponentModel.Win32Exception(Marshal.GetLastWin32Error());
}