Facebook crawler is hitting my server hard and ignoring directives. Accessing same resources multiple times

Question

The Facebook Crawler is hitting my servers multiple times every second and it seems to be ignoring both the Expires header and the og:ttl property.

In some cases, it is

Answer 1

If the FB crawlers ignore your cache headers, adding the "ETag" header could be used in this case to return correct 304 responses and reduce the load of your server.

The first time you generate an image, calculate the hash of that image (for example using md5) as the "ETag" response header. If your server receives a request with the "If-None-Match" header, check if you already have returned that hash. If the answer is yes, return a 304 response. If not, generate the image.

Checking if you already have returned a given hash (while avoiding to generate the image again) means that you'll need to store the hash somewhere... Maybe saving the images in a tmp folder and using the hash as the file name?

More info about "ETag" + "If-None-Match" headers.