I am using below command line curl for knowing if my site supports compressing and caching
curl --head --compress http://www.mysite.com
it retur
In some case I had, faking the agent solved this problem, by using:
curl -A "Mozilla/4.0"
Similarly using libcurl C-API:
libcurl C-API
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/4.0");