Istio envoy 504 gateway timeouts after 15 seconds for outbound connections

Question

Background:

I am using istio 1.2.5

I have deployed istio using helm default profile from the istio documentation by enablin

Answer 1

It seems 15 seconds is a default timeout value. Whether it is Istio or Envoy which sets that, I have yet to read further.

But, there's a couple of reported issue such as #1888 (Istio 0.2.12 and Kubernetes 1.8.1) and #6860 which was discussed to be very similar to your issue.:

INGRESS > PUBLICSERVICE (Timeout 60 works)

PUBLICSERVICE > BACKENDSERVICE (Timeout 60 works)

but

INGRESS > PUBLICSERVICE > BACKENDSERVICE (i get timeout after default 15seconds)

You can read through the discussion of 6860, but in summary, they have introduced a fix starting 1.0.0 so that the timeout can be set greater than 15 seconds. To do that, you need to write the timeout in your VirtualServices. For example, for the internal service set 60 seconds, for your external service set to 40 seconds so that there is enough allowance.

Please check and see if it helps.

Answer 2

After lot of hit and trial we got this working.

Reference: https://github.com/istio/istio/issues/16915#issuecomment-529210672

So for any outbound traffic no matter if you have egress gateway or not default timeout is 15 seconds. So, if you want to increase that you need to have a ServiceEntry and a VirtualService that defines the timeout.

ServiceEntry

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-se-test
spec:
  hosts:
  - slowwly.robertomurray.co.uk
  location: MESH_EXTERNAL
  ports:
  - number: 80
    name: example-http
    protocol: HTTP
  resolution: DNS

VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: slow-ext
spec:
  hosts:
    - slowwly.robertomurray.co.uk
  http:
  - timeout: 30s
    route:
      - destination:
          host: slowwly.robertomurray.co.uk
        weight: 100