how to create a report on filtered items in spring-batch?

Question

I basically want to create a report with the items that were filtered out in every step in my batch. Currently I can see how many were filtered (and skipped) with spring-batch-admin, but I want the list of items so I can take care of them manually.

My step runs over a list of clients and brings data on them from a 3rd party source. I have username+password for every account which I use to get the data. It's possible for a client not to have credentials. I filter those before I run the processor. But a client might have wrong credentials (he miss-typed them) and then I also want to filter it but "differently". I don't want to skip those items because I don't want to set a limit after which my job will fail. Even if all of them were nulls or in-correct, I'd like to continue to the next stage.

My chunk currently consist of a DB reader, DB writer and a CompositeItemProcessor that does:

• Validates that the client has credentials and filters out those who doesn't have.
• Runs the data enhancer (another ItemProcessor) that goes and fetch the data. In case this one gets an error for bad credentials it filters the item too (returns null).

What I want to do is:

• Distinguish between the two "types" of filtering that happened (at the same step). Currently I see all of them sum-ed together in the filterCount field in spring-batch-admin.
• Send a report with the clients that were filtered for each case for manual handling later. I'd also like to aggregate filtered items from a couple of steps and send them together at the end.

What would be the best approach? Can I know what ItemProcessor caused the item to be filtered? I know there are ItemListeners that gets notified after an item is being processed, should I use these somehow? How do I store the results (filtered items) until the end of the job?

I tried to look for a best-practice on this one since I don't think I'm the only one trying to create a report after his job was run, but couldn't find anything useful..

Thanks!

Answer 1

The best way to approach this would be to raise exceptions from within your ItemProcessor(s) and filter those as skip exceptions. Then use a SkipListener to write them to another output. Place your detailed message in the message for the exception so you can include it in your output. Setting the skip limit to something extremely high such as 999999 will prevent your job from failing due to the number of skips.

I have done this very successfully on quite a few spring-batch jobs and it's not an uncommon pattern.

Let me know if you have further questions!

Answer 2

I'm facing a similar situation, and the approach that I've decided to use is inspired by the null object pattern. I filter unwanted items as usual by returning null. If I find an item that I want to still be written, but differently, I return a non-null, but one with a specific state that would be impossible to enter during normal execution. When my writer sees this, it knows to handle it differently. If required, the null-state could be designed to give the writer information about how to interpret it.

This has some pretty obvious drawbacks, but I think the biggest benefit is that it allows you to keep the complexity of your usage of spring's framework down to a minimum.

Also, the fact that you have access to what sounds like plaintext usernames and passwords worries me.