FreeConsole behaviour on Windows 8

Question

On Windows 8, we have an issue with FreeConsole. It seems to close the stdio handles, without shutting the file streams.

This may be a Windows 8 problem, or it could

Answer 1

After disassembling the code for FreeConsole on different Windows versions, I worked out the cause of the problem.

FreeConsole is a remarkably unsubtle function! I really does close a load of handles for you, even if it doesn't "own" those handles (eg HANDLEs owned by stdio functions).

And, the behaviour is different in Windows 7 and 8, and changed again in 10.

Here's the dilemma when coming up with a fix:

• Once stdio has a HANDLE to the console device, there is no documented way to get it to give up that handle, without a call to CloseHandle. You can call close(1) or freopen(stdout) or whatever you like, but if there is an open file descriptor that refers to the console, CloseHandle will be called on it, if you want to switch stdout to a new NUL handle after FreeConsole.
• On the other hand, since Windows 10 there's also no way to avoid FreeConsole calling CloseHandle as well.
• Visual Studio's debugger and the Application Verifier flag the app for calling CloseHandle on an invalid HANDLE. And, they're right, it's really not good.
• So, if you try and "fix up" stdio before the call to FreeConsole then FreeConsole will do an invalid CloseHandle (using its cached handle, and there's no way whatsoever to tell it that handle's gone - FreeConsole no longer checks GetStdHandle(STD_OUTPUT_HANDLE)). And, if you call FreeConsole first, there's no way to fix up the stdio objects without causing them to do an invalid call to CloseHandle.

By elimination, I conclude that the only solution is to use an undocumented function, if the public ones just won't work.

// The undocumented bit!
extern "C" int __cdecl _free_osfhnd(int const fh);
static HANDLE closeFdButNotHandle(int fd) {
  HANDLE h = (HANDLE)_get_osfhandle(fd);
  _free_osfhnd(fd); // Prevent CloseHandle happening in close()
  close(fd);
  return h;
}

static bool valid(HANDLE h) {
  SetLastError(0);
  return GetFileType(h) != FILE_TYPE_UNKNOWN || GetLastError() == 0;
}

static void openNull(int fd, DWORD flags) {
  int newFd;
  // Yet another Microsoft bug! (I've reported four in this code...)
  // They have confirmed a bug in dup2 in Visual Studio 2013, fixed
  // in Visual Studio 2017.  If dup2 is called with fd == newFd, the
  // CRT lock is corrupted, hence the check here before calling dup2.
  if (!_tsopen_s(&newFd, _T("NUL"), flags, _SH_DENYNO, 0) &&
      fd != newFd)
    dup2(newFd, fd);
  if (fd != newFd) close(newFd);
}

void doFreeConsole() {
  // stderr, stdin are similar - left to the reader.  You probably
  // also want to add code (as we have) to detect when the handle
  // is FILE_TYPE_DISK/FILE_TYPE_PIPE and leave the stdio FILE
  // alone if it's actually pointing to disk/pipe.
  HANDLE stdoutHandle = closeFdButNotHandle(fileno(stdout)); 

  FreeConsole(); // error checking left to the reader

  // If FreeConsole *didn't* close the handle then do so now.
  // Has a race condition, but all of this code does so hey.
  if (valid(stdoutHandle)) CloseHandle(stdoutHandle);

  openNull(stdoutRestore, _O_BINARY | _O_RDONLY);
}