HTTP Chunked Encoding. Need an example of 'Trailer' mentioned in SPEC

Question

I am writing an HTTP parser for a transparent proxy. What is stumping me is the Trailer: mentioned in the specs for Transfer-Encoding: chunked. What does it look like?

Normally, a HTTP chunked ends like this.

0\r\n
\r\n

What I am confused about is how to detect the end of the chunk if there is some sort of trailing headers...

UPDATE: I believe that a simple \r\n\r\n i.e. an empty line is enough to detect the end of trailing headers... Is that correct?

Answer 1

Below is a copy of an example trailer I copied from The TCP/IP Guide site.

As we can see, if we want to use trailer header, we need add a "Trailer:header_name" header field with a header name and then add the trailer header entity after chunked body area.

We can add 0 or more trailer headers in a HTTP body per the RFC. Section 4.1.2 of RFC7230 bans the use of following headers in trailer header area:

A sender MUST NOT generate a trailer that contains a field necessary for message framing (e.g., Transfer-Encoding and Content-Length), routing (e.g., Host), request modifiers (e.g., controls and conditionals in Section 5 of RFC7231), authentication (e.g., see RFC7235 and RFC6265), response control data (e.g., see Section 7.1 of RFC7231), or determining how to process the payload (e.g., Content-Encoding, Content-Type, Content-Range, and Trailer).

This means we can use other standard headers and custom headers in trailer header area.

Answer 2

0\r\n
SomeAfterHeader: TheData \r\n
\r\n

In other words, it is sufficient to look for a \r\n\r\n, in layman's terms: a blank line. To detect the end of a chunked transmission. But it is very important that each chunk is read before doing this. Because the chunked data itself can contain blank lines which would erroneously be detected as the end of the stream.

Answer 3

Regarding trailer:

The list of trailing headers should be specified in the Trailer header, as you note.

The BNF in Section 14.40 of RFC 2616 is this:

Trailer  = "Trailer" ":" 1#field-name

Gourley and Totty give this example:

Trailer: Content-Length

(It's odd that they give this example, since Content-Length is explicitly forbidden to be a trailing header in 14.40.)

Shiflett gives this example:

Trailer: Date

Regarding end of message with trailing headers:

The BNF in Section 3.6.1 of RFC 2616 is what you're looking for. Here's part:

Chunked-Body = *chunk
               last-chunk
               trailer
               CRLF
last-chunk   = 1*("0") [ chunk-extension ] CRLF
trailer      = *(entity-header CRLF)

So the last chunk and 2 trailing headers might look like this:

0<CRLF>
Date:Sun, 06 Nov 1994 08:49:37 GMT<CRLF>
Content-MD5:1B2M2Y8AsgTpgAmY7PhCfg==<CRLF>
<CRLF>