Using VS2012 I added the caching feature from the WebRole Properties Caching Tab. Among others, it generated the following XML in web.config:
I got it to work!
I was creating the DataCacheFactory in WebRole OnStart method, I moved it over to Application_Start in Global.asax and it seems to be working.