Go 1.3 Garbage collector not releasing server memory back to system

Question

We wrote the simplest possible TCP server (with minor logging) to examine the memory footprint (see tcp-server.go below)

The server simply accepts connections and do

Answer 1

First, note that Go, itself, doesn't always shrink its own memory space:

https://groups.google.com/forum/#!topic/Golang-Nuts/vfmd6zaRQVs

The heap is freed, you can check this using runtime.ReadMemStats(), but the processes virtual address space does not shrink -- ie, your program will not return memory to the operating system. On Unix based platforms we use a system call to tell the operating system that it can reclaim unused parts of the heap, this facility is not available on Windows platforms.

But you're not on Windows, right?

Well, this thread is less definitive, but it says:

https://groups.google.com/forum/#!topic/golang-nuts/MC2hWpuT7Xc

As I understand, memory is returned to the OS about 5 minutes after is has been marked as free by the GC. And the GC runs every two minutes top, if not triggered by an increase in memory use. So worst-case would be 7 minutes to be freed.

In this case, I think that the slice is not marked as freed, but in use, so it would never be returned to the OS.

It's possible you weren't waiting long enough for the GC sweep followed by the OS return sweep, which could be up to 7 minutes after the final "big" pulse. You can explicitly force this with runtime.FreeOSMemory, but keep in mind that it won't do anything unless the GC has been run.

(Edit: Note that you can force garbage collection with runtime.GC() though obviously you need to be careful how often you use it; you may be able to sync it with sudden downward spikes in connections).

As a slight aside, I can't find an explicit source for this (other than the second thread I posted where someone mentions the same thing), but I recall it being mentioned several times that not all of the memory Go uses is "real" memory. If it's allocated by the runtime but not actually in use by the program, the OS actually has use of the memory regardless of what top or MemStats says, so the amount of memory the program is "really" using is often very overreported.

---

Edit: As Kostix notex in the comments and supports JimB's answer, this question was crossposted on Golang-nuts and we got a rather definitive answer from Dmitri Vyukov:

https://groups.google.com/forum/#!topic/golang-nuts/0WSOKnHGBZE/discussion

I don't there is a solution today. Most of the memory seems to be occupied by goroutine stacks, and we don't release that memory to OS. It will be somewhat better in the next release.

So what I outlines only applies to heap variables, memory on a Goroutine stack will never be released. How exactly this interacts with my last "not all shown allocated system memory is 'real memory'" point remains to be seen.