Deliver stale content after error fetch in Varnish 4 before “probe” marks the server unhealth

Question

I'm using stale-if-error to deliver stale content while the server is marked unhealthy looking at grace and keep object options in vcl_hit.

The question is: Is it possible to deliver a cache object after entering the vcl subroutine vcl_backend_error (fresh request error). Actually, I deliver cached object at vcl_hit but looking at the next diagram, I don't see how it is possible to access the cached object of that request.

Source: https://www.varnish-software.com/book/4.0/chapters/VCL_Basics.html

Answer 1

When using the built-in VCL (see code bellow):

# Built-in 'vcl_hit'.
sub vcl_hit {
    if (obj.ttl >= 0s) {
        return (deliver);
    }

    if (obj.ttl + obj.grace > 0s) {
        return (deliver);
    }

    return (fetch);
}

• If vcl_backend_error is reached by a background / asynchronous backend fetch triggered by a return (deliver) during vcl_hit you don't need to worry. It's just a background thread to update a stalled object. The stalled content has already been delivered to the client.

• If vcl_backend_error is reached by a synchronous backend fetch triggered by a return (fetch) during vcl_hit you don't need to worry too. An error will be delivered to the client, but you have not choice. A stalled object is not available in the Varnish storage.

However, if you have customised vcl_hit to limit grace when the backend is healthy (see VCL example code below), a return (fetch) executed during vcl_hit will be handled as a synchronous backend request. The client will wait for the backend response. If the backend request reaches vcl_backend_error and error will be delivered to the client side. A stalled object is available in the Varnish storage (stalled more than 60 seconds ago in this example), but it's not going to be used.

# Customised 'vcl_hit'.
sub vcl_hit {
    if (obj.ttl >= 0s) {
        return (deliver);
    }

    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 60s > 0s) {
            return (deliver);
        }
    } else {
        if (obj.ttl + obj.grace > 0s) {
            return (deliver);
        }
    }

    return (fetch);
}

If you want to deliver stalled objects when the synchronous backend fetch fails, in this case you need some extra VCL logic. The idea is shown in the code below:

backend fail_be {
    .host = "127.0.0.1";
    .port = "9000";
    .probe = {
        .url = "/give-me-a-non-200-please";
        .interval = 24h;
        .timeout = 1s;
        .window = 1;
        .threshold = 1;
    }
}

sub vcl_recv {
    # Force the non-healthy backend in case of restart because of a previous
    # failed backend fetch. This will force serving stalled content using
    # full grace during 'vcl_hit' (if possible).
    if (req.restarts == 0) {
        unset req.http.X-Varnish-Restarted-5xx;
    } else {
        if (req.http.X-Varnish-Restarted-5xx) {
            set req.backend_hint = fail_be;
        }
    }

    # ...
}

sub vcl_synth {
    # 503 generated for synchronous client requests when abandoning the
    # backend request (see 'vcl_backend_fetch') and not executing a POST.
    if (resp.status == 503 &&
        req.method != "POST" &&
        !req.http.X-Varnish-Restarted-5xx) {
        set req.http.X-Varnish-Restarted-5xx = "1";
        return (restart);
    }

    # ...
}

sub vcl_backend_fetch {
    if (bereq.retries == 0) {
        unset bereq.http.X-Varnish-Backend-5xx;
    } else {
        if (bereq.http.X-Varnish-Backend-5xx) {
            # Jump to 'vcl_synth' with a 503 status code.
            return (abandon);
        }
    }

    # ...
}

sub vcl_backend_response {
    if (beresp.status >= 500 && beresp.status < 600) {
        set bereq.http.X-Varnish-Backend-5xx = "1";
        return (retry);
    }

    # ...
}

sub vcl_backend_error {
    set bereq.http.X-Varnish-Backend-5xx = "1";
    return (retry);
}