Java Excel POI stops after multiple execution by quartz

Question

I'd like to have a couple of insights on this.

I have a program that reads and writes from a database to an excel file. It's execution is based on a timer using Quartz api and triggered every tuesday of the week. The problem is, when i tested it by scheduling it to execute the job every hour, the program suddenly stops after a few executions inside the process of writing the excel file. Here is my write to excel code.

try {
        FileInputStream file = new FileInputStream(excelFile);
        POIFSFileSystem myFileSystem = new POIFSFileSystem(file);
        HSSFWorkbook workbook = new HSSFWorkbook(myFileSystem);
        HSSFSheet worksheet = workbook.getSheetAt(0);
        this.cellStyle00 = workbook.createCellStyle();
        HSSFDataFormat df = workbook.createDataFormat();
        this.cellStyle00.setDataFormat(df.getFormat("00"));

for(int i = 0;i<Access.size();i++){
         AccessorMethods SetGet = (AccessorMethods)
                    InstlibAccessor.get(i);

    HSSFRow row = worksheet.createRow(worksheet.getPhysicalNumberOfRows());
    HSSFCell cell = row.createCell(0);

    cell.setCellValue(new Double(SetGet.getOne()));
    cell.setCellStyle(cellStyle00);


  //other set value codes....

}
FileOutputStream fileOut = new FileOutputStream(fileName + ".xls");
workbook.write(fileOut);
 fileOut.flush();
 fileOut.close(); 

 //catch statements follow
 //end

The command line output and netbeans output does not indicate any error like out of memory, etc etc.. The program does not end.. it just, stops.. Like the jvm is working on a infinite loop... To shed more light on the topic, here is the brief flow of my program.

1. The user executes the scheduler
2. At the desired time the scheduler executes the program (the scheduler and "the program" are two different programs/jar files. The scheduler just calls the jar)
3. The program starts by creating the excel file
4. Then reads database 1. The database contains 80K rows
5. For each row, if a certain condition is met, it reads database 2 and 3
6. It then stores it inside an arraylist object 1000 at a time (Im trying to avoid any memory issues so i store it by batch)
7. Then i write it by batch, 1000 at a time to excel (this is the part it stops)
8. after it completes the reading and writing, it waits until the scheduler calls it again... If it reaches this step I'm a happy programmer =)

Here are some observations I've found;

1. The program usually stops at the 4th to 6th execution of the program (that's after 4 or 6 hours of running the scheduler program non-stop)
2. It stops at a random point of write in excel like at the 34th thousand row or 24th or 15th etc etc etc...
3. This error does not occur when i execute the program without a scheduler. I could manually execute it all day (I did, it was not very fun) without any errors.
4. The output excel file displays 0bytes as size
5. If for example i scheduled it to run every hour and for this hour it stopped. It will still run for the succeeding hours but will stop, and stop at a different point compared to the previous run.

What could have caused this problem. A memory leak perhaps or something more simple?

Additional info

I implemented the Quartz scheduler by importing the class of the other program and run it as a job. here is the code for the trigger

JobDetail job = newJob(ExtractorSchedulerJobUtilTester.class)
            .withIdentity("job1", "group1")
            .build();

    CronTrigger trigger = newTrigger()
            .withIdentity("trigger1", "group1")
            .withSchedule(cronSchedule("0 0/2 * 1/1 * ? *"))
            .build();

    Date ft = sched.scheduleJob(job, trigger);
        sched.start();

and the job

public class ExtractorSchedulerJobUtilTester implements Job {
    public void execute(JobExecutionContext context)
        throws JobExecutionException {
    theProgram program= new theProgram();

    program.main();

    JobKey jobKey = context.getJobDetail().getKey();
    }
}

Is it possible that;

1. The application is maxing out my memory and crashes
2. I'm only using one instance of "program" in my Quartz job which is initialized at the first run of the job and all subsequent executions of the job is referenced from that one instance thus maxing out memory.
3. It's database (AS400) related (I doubt because it stops while it's writing the excel).
4. The computer becomes too tired and decides to take a break.

UPDATE - 12/28/2012

Happy New Year Guys/Gals!!

Sorry it took me some time to get back into this.. (Why bother wasting time on this when the world is gonna end on the 21st. It was bittersweet when it didn't)

I profiled my program with netbeans profiler and got the following figures with the memory analyzer

I noticed in the first graph that my program consumes around 75MB in heap size for every iteration (as indicated by the pink shade). Does this mean that the consumed memory of the program increases by 75mb per iteration? Which, after a few iterations, will consume so much memory that will effect the execution of the program. I'm currently trying to take a heap dump.. I'll post it as soon as I manage to get it running.

Additional Info: I tried using the profiler with only the Quartz running (does not trigger anything) and the system usage is relatively low and the size does not increase for each iteration.

I finally managed to get a heap dump. I took 2 dumps, first is when the first iteration took place and the second is the next iteration. I noticed that there is a big difference between the instances of two of my classess as indicated below

Thanks!

Answer 1

After a lot of cursing, praying and searching I think I have found a possible solution to this. What I did was add System.gc(); at the end of my Quartz job class. Thus, calling the garbage collection everytime the program finishes the job. It's only a possible solution and not a concrete answer because I'm still eating up a lot of heap memory (I believe there's still some memory leak somewhere in the chaos of my code). But, with the System.gc(); I'm consuming considerably less. I'm just not sure how this happens. Logically I would think that GC will only effect the memory allocation and not memory performance of the program. See image below; the top graph is the one with GC while the bottom is the one without.

As you can see the one with the GC is consuming less heap memory than the one without. I assume that the memory usage will still be the same with the GC, but once the GC is called the used heap space will decline. I'll be using this solution for now until a better answer appears.