Calling length on a directory doesn't always return the correct size. You could try to iterate over the file list and add up all file sizes to get the total directory size.
Like this:
long size = 0;
File[] files = cacheDirectory.listFiles();
for (File f:files) {
size = size+f.length();
}