Fast CSV parsing

Question

I have a java server app that download CSV file and parse it. The parsing can take from 5 to 45 minutes, and happens each hour.This method is a bottleneck of the app so it\'

Answer 1

Apart from the suggestions made above, I think you can try improving your code by using some threading and concurrency.

Following is the brief analysis and suggested solution

1. From the code it seems that you are reading the data over the network (most possibly apache-common-httpclient lib).
2. You need to make sure that bottleneck that you are saying is not in the data transfer over the network.
3. One way to see is just dump the data in some file (without parsing) and see how much does it take. This will give you an idea how much time is actually spent in parsing (when compared to current observation).
4. Now have a look at how java.util.concurrent package is used. Some of the link that you can use are (1,2)
5. What you ca do is the tasks that you are doing in for loop can be executed in a thread.
6. Using the threadpool and concurrency will greatly improve your performance.

Though the solution involves some effort, but at the end this will surly help you.