Spring Boot application freezes after a day

a 夏天 提交于 2019-12-23 16:35:41

问题


I have a Spring Boot application running on two servers. It accepts file upload requests (approx 1 KB) via Controller from 1000s of IOT devices.

Additionally there is a TCP Listener which also accepts GPS data as a Stream from these IOT devices.

I have setup the Tomcat max thread count to 3000 on one server and 5000 on a larger VM.

After a day or two my Spring Boot application freezes, i.e. it stops responding to any further API calls.

One reason, I suspect is that the IOT devices are not closing the TCP socket connection after sending the data.

Where could I be going wrong? I am running out of threads to handle new incoming requests?

############################

EDIT 1:

After investigating the thread dump and adding logs during during Socket connection and closure, I have discovered that the IOT device isn't closing connections. Instead the device is creating a new connection every time it wants to send new data without closing the previous one.

Before the app froze, 22500 open connections were automatically closed.

How can I handle this behavior? Should I close the connections forcefully after a certain timeout?


回答1:


You can mitigate the idle connections problem by hiding your service behind an Nginx or HAProxy which close idle connections by default after a given timeout.

Another option is to set server.connection-timeout in application properties (see server properties), but this appears to be applicable to HTTP only.



来源:https://stackoverflow.com/questions/44486788/spring-boot-application-freezes-after-a-day

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!