Google Load-balancer randomly failing requests to backend

Question

I had done simple setup:
Backend(website): 1 machine having tomcat installed and serving on 8080.
webapps folder of tomcat has two folder

1 withGoogleRule1
       |--test.html
2 withoutGoogleRule1
       |--test.html

Now i have a google-load-balancer setup with below settings

Now Backend machine direct ip is http://104.197.37.171:8080/withoutGoogleRule1/test.html

Load Balancer url is http://107.178.240.39:8080/withoutGoogleRule1/test.html

PROBLEM:

No matter how many time we refresh url for direct machine, all requests always servers correctly.

But if you access url using load balancer 1 out of 5 time it shows

The requested URL /withoutGoogleRule1/test.html was not found on this server.

Details:

Youtube Video showing the failure of request

PS:

Instance group details:

LoadBalancer backend details

Answer 1

If your Tomcat was listening only on 8080 instead of 80, then the instance group, backend service configs were problematic. In particular, the health check was configured to check port 80, which would make the load balancer think the backend service was not healthy and return 404.

In addition, you can check Tomcat log to see how many requests were received and what were the responses.

Edit: Since you confirmed another process was listening on 80. The reason for 404 NOT FOUND could be, the requests were dispatched internally to 80. You can verify this theory by hosting a page on both 80 and 8080, e.g. /a.html, but the content is different, one is "i'm 80", the other is "i'm 8080". This way, you can know exactly what happened. Checking the log of the process listening on 80 can also help.