App Engine instance memory constantly increasing

Question

I'd expect the memory usage of my app engine instances (Python) to be relatively flat after an initial startup period. Each request to my app is short lived, and it seems all memory usage of single request should be released shortly afterwards.

This is not the case in practice however. Below is a snapshot of instance memory usage provided by the console. My app has relatively low traffic so I generally have only one instance running. Over the two-day period in the graph, the memory usage trend is constantly increasing. (The two blips are where two instances were briefly running.)

I regularly get memory exceeded errors so I'd like to prevent this continuous increase of memory usage.

At the time of the snapshot:

• Memcache is using less than 1MB
• Task queues are empty
• Traffic is low (0.2 count/second)

I'd expect the instance memory usage to fall in these circumstances, but it isn't happening.

Because I'm using Python with its automatic garbage collection, I don't see how I could have caused this.

Is this expected app engine behavior and is there anything I can do to fix it?

Answer 1

I found another answer that explains part of what is going on here. I'll give a summary based on that answer:

1. When using NDB, entities are stored in a context cache, and the context cache is part of your memory usage.

2. From the documentation, one would expect that memory to be released upon the completion of an HTTP request.

3. In practice, the memory is not released upon the completion of the HTTP request. Apparently, context caches are reused, and the cache is cleared before its next use, which can take a long time to happen.

For my situation, I am adding _use_cache=False to most entities to prevent them from being stored in the context cache. Because of the way my app works, I don't need the context caches for these entities, and this reduces memory usage.

The above is only a partial solution however!

Even with caching turned off for most of my entities, my memory usage is still constantly increasing! Below is snapshot over a 2.5 day period where the memory continuously increases from 36 MB to 81 MB. This is over the 4th of July weekend with low traffic.