Django Response always Chunked with text/html cannot set Content-Length

问题

In my Django Application's views.py , I return an HttpResponse object after attempting to set the following HTTP Header fields:

# Create a Response Object with the content to return 
response = HttpResponse("%s"%(output_display),mimetype='text/html')
response['Cache-Control'] = 'must-revalidate, max-age=20'
response['Vary'] = 'Accept-Encoding'
response['Transfer-Encoding'] = 'gzip'
#response['Content-Encoding'] = 'gzip'
response['Connection'] = 'close'
#response['Content-Type'] = 'text/html'
response['Content-Length'] = '%s'%(len(output_display))
return response

I then capture the output using the Live HTTP Headers plugin with FireFox, and it looks like:

HTTP/1.1 200 OK
Date: Sun, 10 Mar 2013 14:55:09 GMT
Server: Apache/2.2.22 (Ubuntu)
Transfer-Encoding: gzip, chunked   <---------- Why 'chunked'?
Vary: Accept-Encoding
Connection: close
Cache-Control: must-revalidate, max-age=20
Content-Encoding: gzip
Content-Type: text/html <---------------------- No Content-Length even though I set it?
X-Pad: avoid browser bug

I am trying to cache using Apache2's mem_cache, so I need the Content-Length to be set and cannot have 'chunked' for Transfer-Encoding.

My Apache2 mem_cache.conf looks like ( large numbers just for testing ):

<IfModule mod_mem_cache.c>
        CacheEnable mem /
        MCacheSize 10000
        MCacheMaxObjectCount 10000000
        MCacheMinObjectSize 1
        MCacheMaxObjectSize 10000000
        MCacheMaxStreamingBuffer 10000000
</IfModule>

But even though I explicitly set the Content-Length and Transfer-Encoding in my response code, 'chunked' is inserted automatically and therefore my Content-Length is not honored. Why is this? How can I fix this to get the desired response? Thanks -

回答1:

I came across a similar issue recently with a mod_wsgi application; I was trying to update an apache configuration that was using its built-in disk cache, to use socache/memcache instead.

The disk cache was working, but switching to memcache or shmcb didn't work. If I issued a request for a resource I wanted cached, it wouldn't store it in the cache (CacheDetailHeader is helpful for this). Checking the logs at debug, I found the message:

[Wed Dec 05 18:52:16.571002 2018] [cache_socache:debug] \
[pid 884:tid 140422596777728] mod_cache_socache.c(389): \
[client 127.0.0.1:56576] AH02346: URL 'http://127.0.1.1:80/cacheme/c?' \
had no explicit size, ignoring, referer: http://127.0.0.1/

It seems that socache doesn't like objects that don't have explicit sizes. I tried setting the newer, socache equivalents of those mod_memcache settings to sufficiently large values: CacheSocacheMaxSize and CacheSocacheReadSize.

I know that the Content-Length header was being set and made it through to somewhere; it showed up in the mod_wsgi logs when I deliberately miscalculated it.

A few things I found:

Don't set Transfer-Encoding header yourself, as this is forbidden by the WSGI specification:

Who set the Transfer-Encoding: chunked header?
Even though you're setting the Content-Length header yourself, it's also being gzipped by apache. This changes the length; when Apache doesn't know what the length will be, it switches to chunked and removes the Content-Length header.

I found that with:

Content-Type: text/html
Content-Length set to my utf-8 encoding size

set in the python/mod_wsgi application, and:

SetEnv no-gzip 1

set in the apache configuration, that the object made it into a shmcb cache.

It looks like when apache gzips an object, it changes the headers to that it isn't accepted by socache.

I looked around for ways to make them compatible, but couldn't find too much on this issue. There is some mention of reordering the cache/deflate filters in the mod_cache documentation:

https://httpd.apache.org/docs/2.4/mod/mod_cache.html#finecontrol

This worked if I put in a directive to reorder the cache/deflate filters:

# within a directory
SetOutputFilter CACHE;DEFLATE

Curiously, on a cache miss, the server returned gzipped content, but on a cache hit, the server returned unencoded text/html. This looks odd, but I haven't understood the FilterChain directives well enough to try those out.

I also found some mention of this in a related issue with php/content-length: