Why RegisterClass fails with ERROR_NOT_ENOUGH_MEMORY?

Question

Shortly, my question is, why can WinAPI RegisterClass fail with ERROR_NOT_ENOUGH_MEMORY, when there is a lot of memory free, and what can I do to prevent it?

Background: I'm developing an application (WinSCP FTP/SFTP client) that many people use for automating file transfers. Some are running it every minute, every day, from a Windows Scheduler.

I'm getting a lot of reports that after a certain number of runs the application stops working. The number of runs to trigger the problem does not seem to be exact, but it is in range of tens of thousands, to few hundreds of thousands. Also it seems that the problem occurs only when run under Windows Scheduler, not when run in a regular Windows session. Though I cannot 100% confirm this.

Also all reports seem to be for Windows 2008 R2 + some for Windows 7. Again, that can just be a coincidence.

I was myself able to reproduce the problem on Windows 7. Once the system gets into this state, my application no longer starts in Scheduler's session. But it starts just fine in normal regular session. And some other applications (not necessarily all) start even in Scheduler's session. Also in this state I cannot debug the application, as it does not even load when debugger (or tools such as Process Monitor) is running.

The application is using Embarcadero (former Borland) C++ Builder VCL library. It crashes somewhere in VCL initialization code (my WinMain is not even started) and exits with code 3. Inspecting what the initialization code is doing, I was probably able to identify the code that triggers the crash (though it can just be one of many possible causes).

The culprit seems to be RegisterClass WinAPI function that returns 8 (ERROR_NOT_ENOUGH_MEMORY). VCL code throws an exception when this happens; and as there is no exception handler in place yet, it crashes the app.

I have verified this using a very simple C++ console application developed in VS 2012 (to isolate the problem from C++ Builder and VCL). The core code is:

SetLastError(ERROR_SUCCESS);
fout << L"Registering class" << std::endl;
WNDCLASS WndClass;
memset(&WndClass, 0, sizeof(WndClass));
WndClass.lpfnWndProc = &DefWindowProc;
WndClass.lpszClassName = L"TestClass";
WndClass.hInstance = GetModuleHandle(NULL);
ATOM Atom = RegisterClass(&WndClass);
DWORD Error = GetLastError();
// The Atom is NULL and Error is ERROR_NOT_ENOUGH_MEMORY here

(The full code of the test application is at the end)

Despite the error, it does not seem to be a memory issue. What I verified by allocation of 10 MBs of memory, before and after the RegisterClass call (can be seen in the full test code at the end).

Desperate, I even peeked into Wine implementation of RegisterClass. It can indeed fail with ERROR_NOT_ENOUGH_MEMORY, but only when it fails to allocate memory for the class registration. What is few bytes. And it does allocate the memory using HeapAlloc too. Wine won't fail the RegisterClass for any other reason, with any other error code.

To me it looks like a bug in Windows in the first place. I believe Windows should release all resources allocated by a process, when it exits. So no matter how badly the application is implemented, the previous run should not have any impact on subsequent runs in terms of resources (such as memory). Anyway, I would be glad to find a workaround.

Few more facts: The test system does not run anything special except for standard system processes (about 50 in total). In my case it is VMware virtual machine, though my users obviously see the problem on real physical machines. The previous instances of the process are gone, so it's not like that they are not properly terminated, what would keep system from releasing the resources. There's about 500 MBs of memory free (half of total). There are only about 16000 handles allocated.

---

Full code of the test VS application:

#include "stdafx.h"
#include "windows.h"
#include <fstream>

int _tmain(int argc, _TCHAR* argv[])
{
    std::wofstream fout;
    fout.open(L"log.txt",std::ios::app);

    SetLastError(ERROR_SUCCESS);
    fout << L"Allocating heap" << std::endl;
    LPVOID Mem = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, 10 * 1024 * 1024);
    DWORD Error = GetLastError();
    fout << L"HeapAlloc [" << std::hex << intptr_t(Mem) << std::dec 
         << L"] Error [" << Error << "]" << std::endl;

    // ===== Main testing code begins =====
    SetLastError(ERROR_SUCCESS);
    fout << L"Registering class" << std::endl;
    WNDCLASS WndClass;
    memset(&WndClass, 0, sizeof(WndClass));
    WndClass.lpfnWndProc = &DefWindowProc;
    WndClass.lpszClassName = L"TestClass";
    WndClass.hInstance = GetModuleHandle(NULL);
    ATOM Atom = RegisterClass(&WndClass);
    Error = GetLastError();
    fout << L"RegisterClass [" << std::hex << intptr_t(Atom) << std::dec 
         << L"] Error [" << Error << "]" << std::endl;
    // ===== Main testing code ends =====

    SetLastError(ERROR_SUCCESS);
    fout << L"Allocating heap" << std::endl;
    Mem = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, 10 * 1024 * 1024);
    Error = GetLastError();
    fout << L"HeapAlloc [" << std::hex << intptr_t(Mem) << std::dec 
         << L"] Error [" << Error << "]" << std::endl;

    fout << L"Done" << std::endl;

    return 0;
}

Output is (when run from Windows Scheduler on Windows 7 system brought into the state described above by tens of thousands runs of my application):

Allocating heap
HeapAlloc [ec0020] Error [0]
Registering class
RegisterClass [0] Error [8]
Allocating heap
HeapAlloc [18d0020] Error [0]
Done

Answer 1

1. You may run out of available virtual address space before you run out of RAM (especially with 32-bit processes). This does not seem to be the case here, however.
2. The error may refer to running out of some other resource than actual RAM, for example the atoms. Since ATOM is a 16-bit type, there are only 65536 possible atom values. However, global atoms like the window class ones have an even more limited range - 0xC000 through 0xFFFF, giving you only 0x4000 (16384) max registered classes in theory (likely less in practice).

Check the atom values you get from RegisterClass(). If they're getting close to FFFF before erroring out, that's probably your issue.

EDIT: it seems other people has run into the same issue and have identified the culprit:

There is a serious bug in the VCL that will eat up atoms in the private atom table. Windows has a limited number of private atoms in the private atom table (32767), and this is shared by Windows classes, Windows Messages, Clipboard formats, etc. Every time the controls module is initialized, it creates a new Windows Message:

 ControlAtomString := Format('ControlOfs%.8X%.8X', 
                              [HInstance, GetCurrentThreadID]); 
 ControlAtom := GlobalAddAtom(PChar(ControlAtomString));
 RM_GetObjectInstance := RegisterWindowMessage(PChar(ControlAtomString));

The problem is multiplied by the number of DLL's that the application contains that includes the controls module. If you have 10 dll's, and one application, this code will consume 11 atoms every time it is ran.

When the system is ran out of atoms in the private atom table, no window class can be registered. This means, no windowed programs will be able to open after the private atom table is full.

You can also dump the atom table using WinDbg and check for this pattern yourself.