kdevtmpfsi using the entire CPU

梦想的初衷 提交于 2020-12-03 07:39:30

问题


We are using an EC2(Ubuntu) amazon instance for running Apache.Recently we noticed that there is a process using the entire CPU.

We removed it using the help of the following procedure

[root@hadoop002 tmp]# systemctl status 25177
● session-5772.scope - Session 5772 of user root
   Loaded: loaded (/run/systemd/system/session-5772.scope; static; vendor preset: disabled)
  Drop-In: /run/systemd/system/session-5772.scope.d
           └─50-After-systemd-logind\x2eservice.conf, 50-After-systemd-user-sessions\x2eservice.conf, 50-Description.conf, 50-SendSIGHUP.conf, 50-Slice.conf, 50-TasksMax.conf
   Active: active (abandoned) since Wed 2020-01-22 16:06:01 CST; 1h 21min ago
   CGroup: /user.slice/user-0.slice/session-5772.scope
           ├─19331 /var/tmp/kinsing
           └─25177 /tmp/kdevtmpfsi

Jan 22 16:06:17 hadoop002 crontab[19353]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19354]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19366]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19374]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19375]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19383]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19389]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19390]: (root) LIST (root)
Jan 22 16:06:17 hadoop002 crontab[19392]: (root) REPLACE (root)
Jan 22 16:06:17 hadoop002 crontab[19393]: (root) LIST (root)
[root@hadoop002 tmp]# ps -ef|grep kinsing
root     19331     1  0 16:06 ?        00:00:04 /var/tmp/kinsing
root     25190 23274  0 17:27 pts/0    00:00:00 grep --color=auto kinsing
[root@hadoop002 tmp]# ps -ef|grep kdevtmpfsi
root     25177     1 99 17:27 ?        00:01:47 /tmp/kdevtmpfsi
root     25197 23274  0 17:28 pts/0    00:00:00 grep --color=auto kdevtmpfsi
[root@hadoop002 tmp]# kill -9 19331
[root@hadoop002 tmp]# kill -9 25177
[root@hadoop002 tmp]# rm -rf kdevtmpfsi
[root@hadoop002 tmp]# cd /var/tmp/
[root@hadoop002 tmp]# ll
total 16692
-rw-r--r-- 1 root root        0 Jan 13 19:45 aliyun_assist_update.lock
-rwxr-xr-x 1 root root    13176 Dec 20 02:14 for
-rwxr-xr-x 1 root root 17072128 Jan 19 17:43 kinsing
drwx------ 3 root root     4096 Jan 13 19:50 systemd-private-f3342ea6023044bda27f0261d2582ea3-chronyd.service-O7aPIg
[root@hadoop002 tmp]# rm -rf kinsing

But after a few minutes, It again started automatically. Anyone know how to fix this?


回答1:


The solution mentioned here worked for us. You basically create the files the miner uses, without any rights, so the miner cannot create and use them. https://github.com/docker-library/redis/issues/217

touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing

echo "everything is good here" > /tmp/kdevtmpfsi

echo "everything is good here" > /var/tmp/kinsing`

touch /tmp/zzz

echo "everything is good here" > /tmp/zzz

chmod go-rwx /var/tmp

chmod 1777 /tmp`



回答2:


I face to face with it after i installed and run the Flink Cluster. Seem like we are got a attack by a malware, it trying to use our server's cpu to run the program to mine the coin.

My solution is following steps:

  1. Kill the program is running first:

    • Run htop and then push F9 to kill program. We have to kill kdevtmpfsi and kinsing as well.
  2. Delete malware file which is will be run and using the entire CPU (With my centos 7)

    find / -iname kdevtmpfsi -exec rm -fv {} \;

    find / -iname kinsing -exec rm -fv {} \;

The result should be:

/tmp/kdevtmpfsi is removed
/var/tmp/kinsing is removed
  1. Create a file with same name:

    • touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing
    • echo "kdevtmpfsi is fine now" > /tmp/kdevtmpfsi
    • echo "kinsing is fine now" > /var/tmp/kinsing
    • Now make two files is read-only with following command:

    chattr +i /tmp/kdevtmpfsi chattr +i /var/tmp/kinsing

** You should be reboot your server. If its problem is in a remote server, and you are connecting to it with a specify port, you can change to ssh port to increase security!

Hope it help you!




回答3:


I've struggled with this miner for few days and in my case it was the php-fpm:9000 port exposed.
I guess it possible to inject some code remotly this way.

So if you use docker & php-fpm, do NOT run your container this way:

docker run -v /www:/var/www -p 9000:9000 php:7.4

Remove the port mapping: -p 9000:9000

Don't forget to re-build & restart your containers.

More details here: https://github.com/laradock/laradock/issues/2451#issuecomment-577722571




回答4:


The other answer here is good, and you should do everything mentioned there. But, if the thing keeps coming back, and/or you aren't actually using Docker, you probably have an unpatched RCE vulnerability in phpUnit. The details are here:

https://www.sourceclear.com/vulnerability-database/security/remote-code-execution-rce/php/sid-4487

Our situation was:

  • Docker is not being used at all.
  • We had removed all files related to the miner.
  • We had locked things down using the above touch and chmod commands.
  • The damn thing kept trying to run at seemingly random times.

After you've locked things down with the touch/chmod changes, it can't actually DO anything, but it's still annoying, and that phpUnit vulnerability is a HUGE hole that needs plugging anyway.

Hope this helps.




回答5:


I found all the above solutions helpful but all of them are seems the temporary solution as we need to monitor the instance and if we notice any malicious activity then do the same process again.

I have come across this virus around 1 month back and applied all the solutions above which works fine for the limited period after that, it will come again.

Even I didn't install docker in the system so docker open API port was not an issue.

But there are some open-source software which are the open gate for kinsing.

PhpMailer and Solr have some Remote Code Exec vulnerability cause the whole issue.

The easy solution is to upgrade your Solr version to 8.5.1 and there one more thing you can set as security which will 100% remove the virus and it will be permanent.

Here is the full explanation: https://github.com/amulcse/solr-kinsing-malware




回答6:


Maybe this would be useful for somebody. I've found some other entries of kinsing/kdevtmpfsi:

/etc/kinsing

/usr/lib/systemd/system/bot.service

In bot.service:

ExecStart=/etc/kinsing

I've just followed instructions from this tread, deleted both files, rebooted server.

Hope it will help for somebody. I've spent all day trying to solve it.




回答7:


I found this solution to temporary stop the process from running (not using Docker/Redis, so the hole is most likely in phpunit):

/bin/setfacl -m u:www-data:--- /tmp/kinsing
/bin/setfacl -m u:www-data:--- /tmp/kdevtmpfsi

It will prevent user www-data (who, in my case, was running the process) from executing the script.

Also, you will most likely have a cronjob added under user www-data - remove it and run service cron restart!

Remember, that's a temp fix/hack. You must update the vulnerable software to permanently remove this thread!



来源:https://stackoverflow.com/questions/60151640/kdevtmpfsi-using-the-entire-cpu

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!