Processes launched by a service can't allocate memory

Question

This question was migrated from Super User because it can be answered on Stack Overflow. Migrated 8 years ago.

I have a C# service running as the LocalSystem account which launches numerous other processes depending on what its needs are. This has been going fine for a few months. Just this week, some of the sub-processes are crashing. I've attached a remote debugger to them, and they're failing in memory allocations (C++ new operator returns 0x0), which is the indirect cause of the crash.

Funny thing is, if I RDP into the machine, I can easily launch the process from CMD no problems. Yet when the service launches it, no go.

The machine is running Windows XP SP3. It isn't out of the commit charge is about 80% of physical RAM.

Are there some special limitations of how many processes or how much memory can be used by a service, including processes spawned by that service??

Any other ideas why these processes would be unable to allocate memory.

EDIT:

I've had a good look at the crashing scenario with Procmon from SysInternals, and it doesn't reveal anything (that I can see). It all looks like it's going normal, then suddenly crashes. I can confirm from attaching a remote debugger that it is crashing after dereferencing a null pointer from a c++ new call. This is one of the first objects allocated in the app, it should never fail.

I also discovered that if I enable to services option: Allow services to interact with desktop, then all of the child processes launch correctly. The do, however, appear on the desktop when you connect via RDP and are unfortunately terminated if you log out via RDP = YUK! This still isn't an ideal solution, though - I'd really like to know why the child processes were unable to allocate memory after the 6th child process.

Answer 1

I hope this answer will help somebody in the future... I had the same problem - apps would run fine if windows were allowed to render, but would crash on startup if ran under a service and not allowed to interact with desktop. The solution lies in increasing the size of non-interactive windows station heap in the registry, which was set to 512KB on my machine while interactive windows station heap was 3072KB.

You can change this by going to

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

The value is a long string. You need to change the SharedSection setting which looks something like this:

SharedSection=1024,3072,512

The second number is size of interactive windows station heap and the last one is size of non-interactive windows station heap. If you delete the last number then interactive and non-interactive windows station heaps will be of the same size. That's what I did.

Read the details here: http://support.microsoft.com/kb/184802

Answer 2

Are there some special limitations of how many processes or how much memory can be used by a service, including processes spawned by that service??

Job objects can be used to restrict the memory usage of a process (or group of processes), but something would need to associate the processes in question with that job object.

There is no such job object for service processes.

Consider using the registry to allow you to debug from the start-up of the affected processes: http://msdn.microsoft.com/en-us/library/a329t4ed.aspx

Answer 3

You might find SysInterals Procmon useful for seeing what's going on with your processes.