Add Cache-Control and Expires headers to Azure Storage Blobs

Question

I\'m using Azure Storage to serve up static file blobs but I\'d like to add a Cache-Control and Expires header to the files/blobs when served up to reduce bandwidth costs.

Answer 1

I had to run a batch job on about 600k blobs and found 2 things that really helped:

1. Running the operation from a worker role in the same data center. The speed between Azure services is great as long as they are in the same affinity group. Plus there are no data transfer costs.

2. Running the operation in parallel. The Task Parallel Library (TPL) in .net v4 makes this really easy. Here is the code to set the cache-control header for every blob in a container in parallel:

   // get the info for every blob in the container
   var blobInfos = cloudBlobContainer.ListBlobs(
       new BlobRequestOptions() { UseFlatBlobListing = true });
   Parallel.ForEach(blobInfos, (blobInfo) =>
   {
       // get the blob properties
       CloudBlob blob = container.GetBlobReference(blobInfo.Uri.ToString());
       blob.FetchAttributes();
   
       // set cache-control header if necessary
       if (blob.Properties.CacheControl != YOUR_CACHE_CONTROL_HEADER)
       {
           blob.Properties.CacheControl = YOUR_CACHE_CONTROL_HEADER;
           blob.SetProperties();
       }
   });
   

Answer 2

This might be too late to answer, but recently I wanted to do the same in different manner, where I have list of images and needed to apply using powershell script (of course with the help of Azure storage assembly) Hope someone will find this useful in future.

Complete explanation given in Set Azure blob cache-control using powershell script

Add-Type -Path "C:\Program Files\Microsoft SDKs\Windows Azure\.NET SDK\v2.3\ref\Microsoft.WindowsAzure.StorageClient.dll"

$accountName = "[azureaccountname]"
$accountKey = "[azureaccountkey]"
$blobContainerName = "images"

$storageCredentials = New-Object Microsoft.WindowsAzure.StorageCredentialsAccountAndKey -ArgumentList $accountName,$accountKey
$storageAccount = New-Object Microsoft.WindowsAzure.CloudStorageAccount -ArgumentList $storageCredentials,$true
#$blobClient = $storageAccount.CreateCloudBlobClient()
$blobClient =  [Microsoft.WindowsAzure.StorageClient.CloudStorageAccountStorageClientExtensions]::CreateCloudBlobClient($storageAccount)

$cacheControlValue = "public, max-age=604800"

echo "Setting cache control: $cacheControlValue"

Get-Content "imagelist.txt" | foreach {     
    $blobName = "$blobContainerName/$_".Trim()
    echo $blobName
    $blob = $blobClient.GetBlobReference($blobName)
    $blob.Properties.CacheControl = $cacheControlValue
    $blob.SetProperties()
}

Answer 3

Here's an updated version of Joel Fillmore's answer consuming WindowsAzure.Storage v9.3.3. Note that ListBlobsSegmentedAsync returns a page size of 5,000 which is why the BlobContinuationToken is used.

    public async Task BackfillCacheControlAsync()
    {
        var container = await GetCloudBlobContainerAsync();
        BlobContinuationToken continuationToken = null;

        do
        {
            var blobInfos = await container.ListBlobsSegmentedAsync(string.Empty, true, BlobListingDetails.None, null, continuationToken, null, null);
            continuationToken = blobInfos.ContinuationToken;
            foreach (var blobInfo in blobInfos.Results)
            {
                var blockBlob = (CloudBlockBlob)blobInfo;
                var blob = await container.GetBlobReferenceFromServerAsync(blockBlob.Name);
                if (blob.Properties.CacheControl != "public, max-age=31536000")
                {
                    blob.Properties.CacheControl = "public, max-age=31536000";
                    await blob.SetPropertiesAsync();
                }
            }               
        }
        while (continuationToken != null);
    }

    private async Task<CloudBlobContainer> GetCloudBlobContainerAsync()
    {
        var storageAccount = CloudStorageAccount.Parse(_appSettings.AzureStorageConnectionString);
        var blobClient = storageAccount.CreateCloudBlobClient();
        var container = blobClient.GetContainerReference("uploads");
        return container;
    }

Answer 4

Here's an updated version of Joel Fillmore's answer:

Instead of creating a website and using a WorkerRole, Azure now has the ability to run "WebJobs". You can run any executable on demand on a website at the same datacenter where your storage account is located to set cache headers or any other header field.

1. Create a throw-away, temporary website in the same datacenter as your storage account. Don't worry about affinity groups; create an empty ASP.NET site or any other simple site. The content is unimportant.
2. Create a console program using the code below which works with the updated Azure Storage APIs. Compile it for release, and then zip the executable and all required DLLs into a .zip file.
3. Create a WebJob and upload the .zip file from step #2.
4. Run the WebJob. Everything written to the console is available to view in the log file created and accessible from the WebJob control page.
5. Note the UpdateAzureServiceVersion method. Apparently, by default, Azure storage serves improperly formatted ETags so you may wish to run this code once, for details see: this

The code below runs a separate task for each container, and I'm getting about 70 headers updated per second per container. No egress charges.

using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage.Blob;

namespace AzureHeaders
{
    class Program
    {
        static StorageCredentials storageCredentials =
            new StorageCredentials("azureaccountname", @"azzureaccountkey");
        private static string newCacheSettings = "public, max-age=7776000"; // 3 months
        private static string[] containersToProcess = { "container1", "container2" };

        static void Main(string[] args)
        {
            var account = new CloudStorageAccount(
                storageCredentials,
                false /* useHttps */);

            CloudBlobClient blobClient = account.CreateCloudBlobClient();

            var tasks = new List<Task>();
            foreach (var container in blobClient.ListContainers())
            {
                if (containersToProcess.Contains(container.Name))
                {
                    var c = container;
                    tasks.Add(Task.Run(() => FixHeaders(c)));
                }
            }
            Task.WaitAll(tasks.ToArray());
        }

        private static async Task FixHeaders(CloudBlobContainer cloudBlobContainer)
        {
            int totalCount = 0, updateCount = 0, errorCount = 0;

            Console.WriteLine("Starting container: " + cloudBlobContainer.Name);
            IEnumerable<IListBlobItem> blobInfos = cloudBlobContainer.ListBlobs(useFlatBlobListing: true);

            foreach (var blobInfo in blobInfos)
            {
                try
                {
                    CloudBlockBlob blockBlob = (CloudBlockBlob)blobInfo;
                    var blob = await cloudBlobContainer.GetBlobReferenceFromServerAsync(blockBlob.Name);
                    blob.FetchAttributes();

                    // set cache-control header if necessary
                    if (blob.Properties.CacheControl != newCacheSettings)
                    {
                        blob.Properties.CacheControl = newCacheSettings;
                        blob.SetProperties();
                        updateCount++;
                    }
                }
                catch (Exception ex)
                {
                    // Console.WriteLine(ex.Message);
                    errorCount++;
                }
                totalCount++;
            }
            Console.WriteLine("Finished container: " + cloudBlobContainer.Name + 
                ", TotalCount = " + totalCount + 
                ", Updated = " + updateCount + 
                ", Errors = " + errorCount);
        }

        // http://geekswithblogs.net/EltonStoneman/archive/2014/10/09/configure-azure-storage-to-return-proper-response-headers-for-blob.aspx
        private static void UpdateAzureServiceVersion(CloudBlobClient blobClient)
        {
            var props = blobClient.GetServiceProperties();
            props.DefaultServiceVersion = "2014-02-14";
            blobClient.SetServiceProperties(props);
        }
    }
}

Answer 5

Latest CloudBerry Explorer now supports Cache-Control: http://www.cloudberrylab.com/forum/default.aspx?g=posts&t=3047

Answer 6

The latest version of Cerebrata Cloud Storage Studio, v2011.04.23.00, supports setting cache-control on individual blob objects. Right click on the blob object, choose "View/Edit Blob Properties" then set the value for the Cache-Control attribute. (e.g. public, max-age=2592000).

If you check the HTTP headers of the blob object using curl, you'll see the cache-control header returned with the value you set.