ehcache persist to disk issues

Question

I want to do something with ehcache in Java that I think should be extremely simple, but I've spent enough time frustrating myself with the docs...

1. Write a value to a disk persistent cache. Shut down.

2. Start up again and read that value.

Here is my Java function:

private static void testCacheWrite() {

  // create the cache manager from our configuration
  URL url = TestBed.class.getClass().getResource("/resource/ehcache.xml");
  CacheManager manager = CacheManager.create(url);
  // check to see if our cache exits, if it doesn't create it
  Cache testCache = null;
  if (!manager.cacheExists("test")) {
    System.out.println("No cache found. Creating cache...");
    int maxElements = 50000;
    testCache = new Cache("test", maxElements,
      MemoryStoreEvictionPolicy.LFU, true, null, true, 60, 30,
      true, Cache.DEFAULT_EXPIRY_THREAD_INTERVAL_SECONDS, null);
    manager.addCache(testCache);
    // add an element to persist
    Element el = new Element("key", "value");
    testCache.put(el);
    testCache.flush();
    System.out.println("Cache to disk. Cache size on disk: " +
      testCache.getDiskStoreSize());
  } else {
    // cache exists so load it
    testCache = manager.getCache("test");
    Element el = testCache.get("key");
    if (null == el) {
      System.out.print("Value was null");
      return;
    }
    String value = (String) el.getObjectValue();
    System.out.println("Value is: " + value);
  }
  manager.shutdown();
}

And here is my cache configuration (ehcache.xml):

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="../config/ehcache.xsd">
  <diskStore path="C:/mycache"/><!-- java.io.tmpdir -->
  <defaultCache
    maxElementsInMemory="10000"
    eternal="true"
    timeToIdleSeconds="120"
    timeToLiveSeconds="120"
    overflowToDisk="true"
    maxElementsOnDisk="10000000"
    diskPersistent="true"
    diskExpiryThreadIntervalSeconds="120"
    memoryStoreEvictionPolicy="LRU" />
</ehcache>

Even though I see test.index and test.data files on disk after the first run, output from this function is always the following (it never seems to load the cache from disk):

No cache found. Creating cache...
Cache to disk. Cache size on disk: 2

I must be doing something dumb here, but I 'm not sure what!

Answer 1

Okay, well what I did to fix this was configure my cache using the configuration file. Here is the updated config:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:noNamespaceSchemaLocation="../config/ehcache.xsd">

    <diskStore path="C:/mycache" />

    <defaultCache
        maxElementsInMemory="10000" 
        eternal="true"
        timeToIdleSeconds="120" 
        timeToLiveSeconds="120" 
        overflowToDisk="true"
        maxElementsOnDisk="10000000" 
        diskPersistent="true"
        diskExpiryThreadIntervalSeconds="120" 
        memoryStoreEvictionPolicy="LRU" />

    <cache 
        name="test" 
        maxElementsInMemory="500" 
        eternal="true"
        overflowToDisk="true" 
        timeToIdleSeconds="300" 
        timeToLiveSeconds="600"
        diskPersistent="true" 
        diskExpiryThreadIntervalSeconds="1"
        memoryStoreEvictionPolicy="LFU" />

</ehcache>

So basically I didn't use the constructor to define the cache.

I suppose this will work, but I still wonder why programatically defined caches can't persist on disk (especially since they are still written to disk!).

Thanks for the comments guys.

Answer 2

After spending some quality time with the debugger, I believe I have an answer for the OP.

The issue (at least from what I have seen) centers around the non-clustered disk cache files and how they get read back in. In the file net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.java, the method:

public DiskPersistentStorageFactory(Ehcache cache, String diskPath) {
    super(getDataFile(diskPath, cache), cache.getCacheConfiguration().getDiskExpiryThreadIntervalSeconds(),
            cache.getCacheConfiguration().getDiskSpoolBufferSizeMB(), cache.getCacheEventNotificationService(), false);

    indexFile = new File(getDataFile().getParentFile(), getIndexFileName(cache));
    flushTask = new IndexWriteTask(indexFile, cache.getCacheConfiguration().isClearOnFlush());

    if (!getDataFile().exists() || (getDataFile().length() == 0)) {
        LOG.debug("Matching data file missing (or empty) for index file. Deleting index file " + indexFile);
        indexFile.delete();
    } else if (getDataFile().exists() && indexFile.exists()) {
        if (getDataFile().lastModified() > (indexFile.lastModified() + TimeUnit.SECONDS.toMillis(1))) {
            LOG.warn("The index for data file {} is out of date, probably due to an unclean shutdown. " 
                    + "Deleting index file {}", getDataFile(), indexFile);
            indexFile.delete();
        }
    }

    diskCapacity = cache.getCacheConfiguration().getMaxElementsOnDisk();
    memoryCapacity = cache.getCacheConfiguration().getMaxElementsInMemory();
    memoryPolicy = determineEvictionPolicy(cache.getCacheConfiguration());
}

checks the timestamps on the data files. The problem I am seeing is that no matter how I end up shutting down the cache/manager, the files are never get synchronized properly. My quick and dirty workaround was to adjust the time of the data file to be just past the timestamp on the index file:

File index = new File( path, name + ".index" );
File data  = new File( path, name + ".data"  );

data.setLastModified( index.lastModified() + 1 );

Granted, this is not elegant, but it serves my needs, as our project uses clustered caches, and this allows me to debug standalone with a persistent cache...and without having to actually run Terracotta locally.

One caveat is that for non-clustered caches, I do have to flush() after every put() and remove() in order to keep the disk image fresh, especially when debugging due to the lack of shutdown support when you just "pull the plug".

Answer 3

This took me a while to figure out, but basically what needs to be done here is creating the CacheManager accordingly.

If you create the cache manager and the caches the same way how you created it in the xml it will work.

net.sf.ehcache.CacheManager manager = net.sf.ehcache.CacheManager
        .create(new Configuration().diskStore(
            new DiskStoreConfiguration().path("C:/mycache")
        )
        .cache(new CacheConfiguration()
            .name(testName)
            .eternal(true)
            .maxBytesLocalHeap(10000, MemoryUnit.BYTES)
            .maxBytesLocalDisk(1000000, MemoryUnit.BYTES)
            .diskExpiryThreadIntervalSeconds(0)
            .diskPersistent(true)));

Answer 4

this might be a little late but i had the same problem: what helped was to shutdown the cache manager.

(from the docu: http://ehcache.org/documentation/code-samples#ways-of-loading-cache-configuration)

Shutdown the singleton CacheManager:

CacheManager.getInstance().shutdown();

Shutdown a CacheManager instance, assuming you have a reference to the CacheManager called :

manager.shutdown();

Answer 5

I think you should remove the manager.cacheExists(..) test and simply create your cache using testCache = manager.getCache("test"); instead of using new Cache(..). Even if your cache is diskPersistent, it won't exist until you get it the first time. (At least that's what I think as I'm only using getCache(..) and it does exactly what you are looking for)

Note:

You could also add something like this to make sure the cache exists:

Cache cache = manager.getCache(name);
if (cache == null) {
    throw new NullPointerException(String.format("no cache with name %s defined, please configure it in %s", name, url));
}

Note 2:

If your configuration file is called ehcache.xml, you shouldn't use CacheManager.create(url). Instead use the CacheManager singleton: I think I've confused using CacheManager.create(url) with and using new CacheManager(url). Still, you should use the singleton for ehcache.xml and new CacheManager(url) for anything else.

// ehcache.xml - shared between different invocations
CacheManager defaultManager = CacheManager.getInstance();
// others - avoid calling twice with same argument
CacheManager manager = CacheManager.create(url);

Using CacheManager.create(..) is problematic as it might completely ignore the passed URL if any of the create(..) methods or getInstance() have been called before:

public static CacheManager create(URL configurationFileURL) throws CacheException {
    synchronized (CacheManager.class) {
        if (singleton == null) {
            if (LOG.isDebugEnabled()) {
                LOG.debug("Creating new CacheManager with config URL: " + configurationFileURL);
            }
            singleton = new CacheManager(configurationFileURL);

        }
        return singleton;
    }
}

That's why I wouldn't recommend using any of the CacheManager.create(..) methods. Use CacheManager.getInstance() or new CacheManager(url).

Answer 6

Small hint if your cache on disk stays empty: Be sure your elements in the cache are serializable. ehcache does log if thats not the case but my log settings did not print out these log entries.

Answer 7

I had and resolved a similar issue.

I want to configure ehcache to have a given cache persisting elements on disk. But I want to do it only in local environment (production environment works with a distributed persistence) so I switch the configuration programmatically when the application starts (a web application in my case)

File configurationFile = new File(event.getServletContext().getRealPath(EHCACHE_CONFIG_PATH));    
Configuration configuration = ConfigurationFactory.parseConfiguration(configurationFile);

//...doing other stuff here...

CacheConfiguration cacheConfiguration = configuration.getCacheConfigurations().get("mycachename");
if(localEnvironment){    
    cacheConfiguration.addPersistence(new PersistenceConfiguration().strategy(Strategy.DISTRIBUTED));
}else{
    //siteCacheConfiguration.addPersistence(new PersistenceConfiguration().strategy(Strategy.LOCALRESTARTABLE));
    //deprecated lines..
    siteCacheConfiguration.setDiskPersistent(true);
    siteCacheConfiguration.setOverflowToDisk(true);
}

I had problem with the commented line siteCacheConfiguration.addPersistence(new PersistenceConfiguration().strategy(Strategy.LOCALRESTARTABLE)), in fact Ehcache code (I'm using ehcache-2.6.11) throws an Exception if you use Strategy.LOCALRESTARTABLE without an enterprise version of the jar:

CacheException: You must use an enterprise version of Ehcache to successfully enable enterprise persistence.

Digging into the code I realized that these two (deprecated) lines do the same eluding the entreprise version Exception

siteCacheConfiguration.setDiskPersistent(true);
siteCacheConfiguration.setOverflowToDisk(true);

Remember to add CacheManager.getInstance().shutdown() on the shutdown of the application!

Hope this helps.

Answer 8

I suppose this will work, but I still wonder why programatically defined caches can't persist on disk (especially since they are still written to disk!)

My understanding is that a programmatically created cache (i.e. not declared in ehcache.xml) can use a DiskStore that can itself be persistent but this doesn't mean that this cache will be loaded automatically by the CacheManager uppon restart. Actually, I don't think the previously mentioned files do contain the cache parameters.

But, if you "recreate" the cache programmatically with the same parameters, you'll find the previously cached entries back from the DiskStore.