How to read files in multithreaded mode?

Question

I currently have a program that reads file (very huge) in single threaded mode and creates search index but it takes too long to index in single threaded environment.

<

Answer 1

Your bottleneck is most likely the indexing, not the file reading. assuming your indexing system supports multiple threads, you probably want a producer/consumer setup with one thread reading the file and pushing each line into a BlockingQueue (the producer), and multiple threads pulling lines from the BlockingQueue and pushing them into the index (the consumers).