You're storing all (sub-) files and folder recursively in a list, but with your current code you store the parent folder before you store the children. And so you try to delete the folder before it is empty. Try this code:
if(tempFile.isDirectory()) {
// children first
fetchCompleteList(filesList, folderList, tempFile.getAbsolutePath());
// parent folder last
folderList.add(tempFile.getAbsolutePath());
}