How to split a CSV file into multiple chunks and read those chunks in parallel in Java code

Question

I have a very big CSV file (1GB+), it has 100,000 line.

I need to write a Java program to parse each line from the CSV file to create a body for a HTTP request to s

Answer 1

Reading a single file at multiple positions concurrently wouldn't let you go any faster (but it could slow you down considerably).

Instead of reading the file from multiple threads, read the file from a single thread, and parallelize the processing of these lines. A singe thread should read your CSV line-by-line, and put each line in a queue. Multiple working threads should then take the next line from the queue, parse it, convert to a request, and process the request concurrently as needed. The splitting of the work would then be done by a single thread, ensuring that there are no missing lines or overlaps.