问题
I have a Spring Boot application running on two servers. It accepts file upload requests (approx 1 KB) via Controller from 1000s of IOT devices.
Additionally there is a TCP Listener which also accepts GPS data as a Stream from these IOT devices.
I have setup the Tomcat max thread count to 3000 on one server and 5000 on a larger VM.
After a day or two my Spring Boot application freezes, i.e. it stops responding to any further API calls.
One reason, I suspect is that the IOT devices are not closing the TCP socket connection after sending the data.
Where could I be going wrong? I am running out of threads to handle new incoming requests?
############################
EDIT 1:
After investigating the thread dump and adding logs during during Socket connection and closure, I have discovered that the IOT device isn't closing connections. Instead the device is creating a new connection every time it wants to send new data without closing the previous one.
Before the app froze, 22500 open connections were automatically closed.
How can I handle this behavior? Should I close the connections forcefully after a certain timeout?
回答1:
You can mitigate the idle connections problem by hiding your service behind an Nginx or HAProxy which close idle connections by default after a given timeout.
Another option is to set server.connection-timeout
in application properties (see server properties), but this appears to be applicable to HTTP only.
来源:https://stackoverflow.com/questions/44486788/spring-boot-application-freezes-after-a-day