问题
My cron service keep killed by OS around 2-3 day after start which really frustrating, but seems no one encounter with this situation(nothing learn from google). Can you advice how to trace this issue?
- This behavior is random, no explicit date and time. This time killed on 00:00:01.
- resources is enough.
09:55:01 PM CPU %user %nice %system %iowait %steal %idle
10:05:01 PM all 2.45 0.00 4.04 0.06 0.00 93.45
10:15:01 PM all 1.27 0.00 3.98 0.00 0.00 94.74
10:25:01 PM all 1.46 0.00 3.96 0.01 0.00 94.58
10:35:01 PM all 1.35 0.00 4.06 0.05 0.00 94.55
10:45:01 PM all 1.30 0.00 4.11 0.00 0.00 94.59
10:55:01 PM all 1.26 0.00 4.11 0.00 0.00 94.63
11:05:01 PM all 1.27 0.00 4.11 0.00 0.00 94.61
11:15:01 PM all 1.29 0.00 4.09 0.00 0.00 94.62
11:25:01 PM all 1.29 0.00 4.03 0.00 0.00 94.67
11:35:02 PM all 1.28 0.00 4.01 0.00 0.00 94.71
11:45:01 PM all 1.27 0.00 4.03 0.00 0.00 94.70
11:55:01 PM all 1.27 0.00 4.03 0.00 0.00 94.71
11:59:01 PM all 1.27 0.00 4.04 0.00 0.00 94.68
12:00:01 AM all 1.23 0.00 4.08 0.00 0.00 94.69
Average: all 1.29 0.00 4.12 0.01 0.00 94.58
09:55:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
10:05:01 PM 188444 1702008 90.03 162168 421180 1900536 100.53 1345564 203788 4
10:15:01 PM 188172 1702280 90.05 162168 421184 1902904 100.66 1346432 203604 4
10:25:01 PM 185436 1705016 90.19 162240 421276 1900980 100.56 1349056 203232 0
10:35:01 PM 184816 1705636 90.22 162240 421480 1901204 100.57 1349884 203040 4
10:45:01 PM 184592 1705860 90.24 162240 421484 1903888 100.71 1350236 202816 4
10:55:01 PM 185644 1704808 90.18 162244 421488 1900008 100.51 1349720 202700 4
11:05:01 PM 185392 1705060 90.19 162244 421492 1899364 100.47 1349668 202580 4
11:15:01 PM 184740 1705712 90.23 162244 421500 1903640 100.70 1350164 202524 4
11:25:01 PM 184844 1705608 90.22 162244 421504 1901208 100.57 1350088 202416 4
11:35:02 PM 184656 1705796 90.23 162244 421508 1902928 100.66 1350940 202320 4
11:45:01 PM 184440 1706012 90.24 162244 421516 1902392 100.63 1350564 202308 4
11:55:01 PM 184968 1705484 90.22 162244 421524 1899148 100.46 1350128 202292 4
11:59:01 PM 185064 1705388 90.21 162244 421528 1900400 100.53 1350100 202268 4
12:00:01 AM 182740 1707712 90.33 162244 421532 1907832 100.92 1352400 202244 4
Average: 186998 1703454 90.11 162546 426534 1900233 100.52 1322833 224168 5
- no error mentioned in syslog.
Dec 18 23:59:01 localhost CRON[32201]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 60 2)
Dec 19 00:00:01 localhost CRON[1147]: (root) CMD (/tmp/.X17-unix/.rsync/c/aptitude>/dev/null 2>&1)
Dec 19 00:00:01 localhost CRON[1149]: (wp) CMD (/var/lib/docker/volumes/root_wp_data/_data/.bashtemp/a/upd>/dev/null 2>&1)
Dec 19 00:00:01 localhost CRON[1148]: (root) CMD (/usr/lib/armbian/armbian-apt-updates)
Dec 19 00:00:01 localhost CRON[1150]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs)
Dec 19 00:00:01 localhost CRON[1153]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
Dec 19 00:00:01 localhost CRON[1152]: (root) CMD (/root/.nullcache/a/upd>/dev/null 2>&1)
Dec 19 00:00:01 localhost CRON[1158]: (wp) CMD (/tmp/.X19-unix/.rsync/c/aptitude>/dev/null 2>&1)
Dec 19 00:00:01 localhost systemd[1]: cron.service: Main process exited, code=killed, status=9/KILL
Dec 19 00:00:01 localhost systemd[1]: cron.service: Unit entered failed state.
Dec 19 00:00:01 localhost systemd[1]: cron.service: Failed with result 'signal'.
Dec 19 00:58:57 localhost dhclient[2585]: DHCPREQUEST of 192.168.1.11 on wlan0 to 192.168.1.1 port 67 (xid=0x5584c9ee)
Dec 19 02:08:42 localhost dhclient[2585]: message repeated 312 times: [ DHCPREQUEST of 192.168.1.11 on wlan0 to 192.168.1.1 port 67 (xid=0x5584c9ee)]
- service log
● cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Thu 2019-12-19 00:00:01 HKT; 22h ago
Docs: man:cron(8)
Process: 2176 ExecStart=/usr/sbin/cron -f $EXTRA_OPTS (code=killed, signal=KILL)
Main PID: 2176 (code=killed, signal=KILL)
Tasks: 17
Memory: 125.9M
CPU: 10h 42min 23.763s
CGroup: /system.slice/cron.service
├─ 1185 /bin/bash ./run
├─ 1252 /bin/bash ./go
├─ 3863 /bin/bash ./run
├─10646 rsync
├─10647 rsync
├─18403 /bin/bash ./go
├─18774 sleep 3
├─18779 sleep 3
├─18780 sleep 3
├─18782 sleep 3
├─18800 sleep 3
├─19727 /bin/bash ./go
├─24144 /bin/bash ./run
├─24926 /bin/bash ./go
├─26147 /bin/bash ./run
├─29234 /bin/bash ./go
└─30797 /bin/bash ./run
Dec 19 00:00:01 aml CRON[1150]: (root) CMD (/usr/lib/armbian/armbian-truncate-logs)
Dec 19 00:00:01 aml CRON[1142]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 19 00:00:01 aml CRON[1153]: (root) CMD (test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew)
Dec 19 00:00:01 aml CRON[1145]: pam_unix(cron:session): session opened for user wp by (uid=0)
Dec 19 00:00:01 aml CRON[1152]: (root) CMD (/root/.nullcache/a/upd>/dev/null 2>&1)
Dec 19 00:00:01 aml CRON[1143]: pam_unix(cron:session): session closed for user root
Dec 19 00:00:01 aml CRON[1158]: (wp) CMD (/tmp/.X19-unix/.rsync/c/aptitude>/dev/null 2>&1)
Dec 19 00:00:01 aml systemd[1]: cron.service: Main process exited, code=killed, status=9/KILL
Dec 19 00:00:01 aml systemd[1]: cron.service: Unit entered failed state.
Dec 19 00:00:01 aml systemd[1]: cron.service: Failed with result 'signal'.
回答1:
Your account was compromised (I had the similar issue just couple of days ago). Look at this for quik solution: crond64/tsm virus in Ubuntu.
And also very good detailed report (in Russian): Тайная жизнь Linux сервера или веерная брутфорс атака на подсистему SSH, translated as something like: The secret life of a Linux server or fan brute force attack on the SSH subsystem.
来源：https://stackoverflow.com/questions/59411428/cron-killed-by-system