PHP-MySQLi connection randomly fails with “Cannot assign requested address”

Question

Since about 2 weeks I\'m dealing with one of the weirdest problems in LAMP stack. Long story short randomly connection to MySQL server is failing with error message:

<         


        

Answer 1

With Vicidial I have run into the same problem frequently, due to the kind of programming used, new MYSQL connections have to be established (very) frequently from a number of vicidial components, we have systems hammering the db server with over 10000 connections per second, most of which are serviced within a few ms and which are closed within a second or less. From experience I can tell you that in a local network, with close to no lost packages, tcp_fin_timeout can be reduced all the way down to 3 with no problems showing up.

Typical linux commands to diagnose if connections waiting to be closed is your problem are:

netstat -anlp | grep :3306 | grep TIME_WAIT -wc

which will show you the number of connections that are waiting to be closed completely.

netstat -nat | awk {'print $5'} | cut -d ":" -f1 | sort | uniq -c | sort -n

which will show the connections per connected host, allowing you to identify which other host is folding your system if there are multiple candidates.

To test the fix you can just

cat /proc/sys/net/ipv4/tcp_fin_timeout
echo "3" > /proc/sys/net/ipv4/tcp_fin_timeout

which will temporarily set the tcp_fin_timeout to 3 sec and tell you how many seconds it was before, so you can revert to the old value for testing.

As a permanent fix I would suggest you add the following line to /etc/sysctl.conf

net.ipv4.tcp_fin_timeout=3

Within a good local network with should not cause any trouble, if you do run into problems e.g. because of packet loss, you can try

net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_fin_timeout=10

Wiche allows more time for the connection to close and tries to reuse same ip:port combinations for new connections to the same host:service combination.

OR

net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=10

Which will even more aggressively try to reuse connections, what can however create new problems with other applications for example with your webserver. So you should try the simple solution first, in most cases it will already fix your problem without any bad side effects!

Good Luck!