Django memory usage going up with every request

Question

I moved my first Django project from DjangoEurope to Webfaction, and that started an issue looking like a memory leak. With every single request memory usage of the server process goes up about 500kb. It never goes down. This goes on until Webfaction kills it for using too much memory.

I can clearly see this when I refresh the Django's admin interface in my browser (although this happens with every single page, not only with admin interface - I though admin interface would be a nice test case, because there is no my code directly there). With every browser reload the memory usage goes up couple hundreds kilobytes.

I test the memory using a command suggested by Webfaction:

ps -u publica -o rss,etime,pid,command

More about my setup:

• Django 1.1 (stable)
• Default Webfaction Django setup using Apache and mod_wsgi
• DEBUG set to False
• MySQLdb 1.2.2 from Webfaction, but after hearing it had some problems I tried version 1.2.3c. Didn't help.

Edit: I created an empty Django project - default Django configuration plus django.contrib.admin and fresh empty database (tried both with mysql and postgresql). I started reloading Django admin in my browser and watched memory usage. At first I saw the problem occurring - memory usage grew after every reload. But then it stabilized and stopped growing. That's consistant with how my original project behaved on Django Europe. Unfortunately on Webfaction it never seems to stabilize (or at least not within limits of memory available to my account). Any advice?

Answer 1

I'm afraid I haven't got any definite answers. Graham Dumpleton's tips were most helpfull, but unfortunately he didn't make an answer (just comments), so there is no way to accept his response.

Although I still haven't fully resolved the issue, here are some basic tips for other people having similar problems:

• Read Webfaction's documentation:
  • Reducing Memory Usage in general
  • Reducing mod_wsgi Memory Consumption
  • Reducing Django Memory Consumption
• Make sure the DEBUG setting is set to False
• Don't use mod_python, use mod_wsgi
• Make sure you use the most recent version od mod_wsgi (Webfaction tends to install older versions)
• Don't use Django to serve static content
• Try running mod_wsgi in a daemon mode (Webfaction installs it in embedded mode by default) [thanks Graham Dumpleton for the tip]
• If you run in embeded mode, you can specify "inactivity-timeout=[seconds]" option. It will restart the process after [seconds] of inactivity, helping with increased memory usage. Read this forum post for detailed instructions.
• This script will help you monitor your memory usage easier, and more precisely [thanks ClaudioA for the tip]

Answer 2

We had a similar issue on Webfaction, but it turned out it wasn't because of them at all. There's a bug in Django about high memory usage when using the sitemap with lots of elements: http://code.djangoproject.com/ticket/11572

When we removed the sitemap it doesn't shoot up 90 Mb at the time any longer. Just thought I should mention it since it took a long time for use to troubleshoot.

Answer 3

I have the same problems with webfaction.

The method I use, and which webfaction told me I should keep using, is run a cron job that checks the memory every 5 minutes or so, and restarts any apps that are getting out of control.

Out of 4 python apps on webfaction, I average 4 restarts per day.

Answer 4

I'd suggest not to guess.

Take a look at http://code.google.com/p/django-dowser/, it is a very useful app for detecting memory leaks and figuring out what parts of your code are responsible for memory consumption.

Answer 5

Check if the in-process memory cache backend is enabled, if yes, that could be the problem (new cache entries at each request).

Answer 6

I've had issues with memory on Webfaction too - they didn't really crop up until I added my fifth application though. I tried a few tweaks to my apache configs, but what finally worked for me was just switching over to mod_wsgi.

Answer 7

Are both mod_python and mod_wsgi modules loaded into apache?

I know mod_wsgi doesn't like having mod_python in its building. Check its not loaded.

Answer 8

• Did you check these tips? -> http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory-usage-down

• We have deploy an app in WebFaction and we are not having those issues.

• Are you serving your static files as they recommend?

• BTW this could be useful too: http://fightingrabbits.com/archives/198