Automatically kill process that consume too much memory or stall on linux

问题

I would like a "system" that monitors a process and would kill said process if:

the process exceeds some memory requirements
the process does not respond to a message from the "system" in some period of time

I assume this "system" could be something as simple as a monitoring process? A code example of how this could be done would be useful. I am of course not averse to a completely different solution to this problem.

回答1:

For the first requirement, you might want to look into either using ulimit, or tweaking the kernel OOM-killer settings on your system.

Monitoring daemons exist for this sort of thing as well. God is a recent example.

回答2:

I wrote a script that runs as a cron job and can be customized to kill problem processes:

#!/usr/local/bin/perl

use strict;
use warnings;
use Proc::ProcessTable;

my $table = Proc::ProcessTable->new;

for my $process (@{$table->table}) {
    # skip root processes
    next if $process->uid == 0 or $process->gid == 0;

    # skip anything other than Passenger application processes
    #next unless $process->fname eq 'ruby' and $process->cmndline =~ /\bRails\b/;

    # skip any using less than 1 GiB
    next if $process->rss < 1_073_741_824;

    # document the slaughter
    (my $cmd = $process->cmndline) =~ s/\s+\z//;
    print "Killing process: pid=", $process->pid, " uid=", $process->uid, " rss=", $process->rss, " fname=", $process->fname, " cmndline=", $cmd, "\n";

    # try first to terminate process politely
    kill 15, $process->pid;

    # wait a little, then kill ruthlessly if it's still around
    sleep 5;
    kill 9, $process->pid;
}

http://blog.endpoint.com/2012/08/automatically-kill-process-using-too.html

回答3:

To limit memory usage of processes, check /etc/security/limits.conf

回答4:

If you want to set up a fairly comprehensive monitoring system, check out monit. It can be very (VERY VERY VERY VERY) chatty at times, but it will do a lot of monitoring, restart services, alert you, etc.

That said, don't be surprised if you're getting dozens of e-mails a day until you get used to configuring it and telling it what not to bug you about.

回答5:

Try Process Resource Monitor for a classic, easy-to-use process monitor. Code available under the GPL.

There's a few other monitoring scripts there you might find interesting too.

回答6:

Are the monitored processes ones you're writing, or just any process?

If they're arbitrary processes then it might be hard to monitor for responsiveness. Unless the process is already set up to handle and respond to events that you can send it, then I doubt you'll be able to monitor them. If they're processes that you're writing, you'd need to add some kind of message handling that you can use the check against.

来源：https://stackoverflow.com/questions/187804/automatically-kill-process-that-consume-too-much-memory-or-stall-on-linux

标签

Linux

project-management

process