Reading a file by multiple threads

问题

I have a 250Mb file to be read. And the application is multi threaded. If i allow all threads to read the file the memory starvation occurs. I get out of memory error.

To avoid it. I want to have only one copy of the String (which is read from stream) in memory and i want all the threads to use it.

while (true) {
    synchronized (buffer) {
        num = is.read(buffer);
            String str = new String(buffer, 0, num);

    }
    sendToPC(str);
}

Basically i want to have only one copy of string when all thread completed sending, i want to read second string and so on.

回答1:

Why multiple threads? You only have one disk and it can only go so fast. Multithreading it won't help, almost certainly. And any software design that relies on having an entire file in memory is seriously flawed in the first place.

Suppose you define your problem?

回答2:

I realize this is kind of late, but I think what you want here is to use the map function in the FileChannel class. Once you map a region of the file into memory, then all of your threads can read or write to that block of memory and the OS will synchronizes that memory region with the file periodically (or when you call MappedByteBuffer.load()), and if you want each thread to work with a different part of the file, then you can assign several maps each mapping a specific region of the file and just use one map per thread.

see the javadoc for FileChannel, RandomAccessFile, and MappedByteBuffer

回答3:

Could you directly use streams instead of completely reading the file in to memory?

回答4:

You can register all threads as callbacks in the File reading class. SO have something like an array or list of classes implementing an interface StringReaderThread which has the method processString(String input). After reading each line from the file, iterate over this array/list and call processString() on all the threads this way. Would this solve your problem?

来源：https://stackoverflow.com/questions/5165226/reading-a-file-by-multiple-threads