Automatically kill process that consume too much memory or stall on linux

Question

I would like a "system" that monitors a process and would kill said process if:

• the process exceeds some memory requirements
• the process does not respond to a message from the "system" in some period of time

I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem.

Answer 1

For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.

Monitoring daemons exist for this sort of thing as well. God is a recent example.

Answer 2

I wrote a script that runs as a cron job and can be customized to kill problem processes:

#!/usr/local/bin/perl

use strict;
use warnings;
use Proc::ProcessTable;

my $table = Proc::ProcessTable->new;

for my $process (@{$table->table}) {
    # skip root processes
    next if $process->uid == 0 or $process->gid == 0;

    # skip anything other than Passenger application processes
    #next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/;

    # skip any using less than 1 GiB
    next if $process->rss < 1_073_741_824;

    # document the slaughter
    (my $cmd = $process->cmndline) =~ s/\s+\z//;
    print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n";

    # try first to terminate process politely
    kill 15, $process->pid;

    # wait a little, then kill ruthlessly if it's still around
    sleep 5;
    kill 9, $process->pid;
}

http://blog.endpoint.com/2012/08/automatically-kill-process-using-too.html

Answer 3

To limit memory usage of processes, check /etc/security/limits.conf

Answer 4

If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very (VERY VERY VERY VERY) chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.

That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.

Answer 5

Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.

There's a few other monitoring scripts there you might find interesting too.

Answer 6

Are the monitored processes ones you're writing, or just any process?

If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.