upstream sent too big header while reading response header from upstream

前端未结

关注

 8  942

醉梦人生 2020-11-27 09:33

I am getting these kind of errors:

2014/05/24 11:49:06 [error] 8376#0: *54031 upstream sent too big header while reading response header from upstream

8条回答

暖寄归人 (楼主)

2020-11-27 10:05

We ended up realising that our one server that was experiencing this had busted fpm config resulting in php errors/warnings/notices that'd normally be logged to disk were being sent over the FCGI socket. It looks like there's a parsing bug when part of the header gets split across the buffer chunks.

So setting php_admin_value[error_log] to something actually writeable and restarting php-fpm was enough to fix the problem.

We could reproduce the problem with a smaller script:



Raising the buffers made the 502s harder to hit but not impossible, e.g native:

bash-4.1# for it in {30..200..3}; do for size in {100..250..3}; do echo "size=$size iterations=$it $(curl -sv "http://localhost/debug.php?size=$size&iterations=$it" 2>&1 | egrep '^< HTTP')"; done; done | grep 502 | head
size=121 iterations=30 < HTTP/1.1 502 Bad Gateway
size=109 iterations=33 < HTTP/1.1 502 Bad Gateway
size=232 iterations=33 < HTTP/1.1 502 Bad Gateway
size=241 iterations=48 < HTTP/1.1 502 Bad Gateway
size=145 iterations=51 < HTTP/1.1 502 Bad Gateway
size=226 iterations=51 < HTTP/1.1 502 Bad Gateway
size=190 iterations=60 < HTTP/1.1 502 Bad Gateway
size=115 iterations=63 < HTTP/1.1 502 Bad Gateway
size=109 iterations=66 < HTTP/1.1 502 Bad Gateway
size=163 iterations=69 < HTTP/1.1 502 Bad Gateway
[... there would be more here, but I piped through head ...]


fastcgi_buffers 16 16k; fastcgi_buffer_size 32k;:

bash-4.1# for it in {30..200..3}; do for size in {100..250..3}; do echo "size=$size iterations=$it $(curl -sv "http://localhost/debug.php?size=$size&iterations=$it" 2>&1 | egrep '^< HTTP')"; done; done | grep 502 | head
size=223 iterations=69 < HTTP/1.1 502 Bad Gateway
size=184 iterations=165 < HTTP/1.1 502 Bad Gateway
size=151 iterations=198 < HTTP/1.1 502 Bad Gateway


So I believe the correct answer is: fix your fpm config so it logs errors to disk.