Out of memory using Scanner to read large file into memory

问题

The following block of code is throwing a java.lang.OutOfMemoryError exception when I pass a large file into the scanner. What is the best way to fix this? Is the problem in the arraylist or the scanner?

ArrayList rawData = new ArrayList();
Scanner scan = new Scanner(file);

while (scan.hasNext()) {
    String next = scan.next();
        rawData.add(next);
}

回答1:

Increase the java heap size, e.g.

java -Xmx6g myprogram

will set the heap size to 6 gigabytes. Of course there will always be a limit....

回答2:

The main problem is storing on the array list. Also, try to use the bufferReader and just do the processing inside the while statement instead of trying to add it to the arraylist. Here is a simple example.

        File file = new File("C:\\custom_programs\\reminder_list.txt");
        BufferedReader br = new BufferedReader(new FileReader(file));
        String line;
        while ((line = br.readLine()) != null) {
            // do something with line.
            System.out.println(line);
        }
        br.close();

回答3:

Instead of loading all the rows from file into ArrayList, do the operation you want to on each record as soon as you read it. Loading whole file into memory would cause an OOM issue if heapsize is not large enough.

Scanner scan = new Scanner(file);
while (scan.hasNext()) {
    String next = scan.next();
    //do what you want to do on next
}

回答4:

The default separator for a Scanner is whitespace.

public Scanner(ReadableByteChannel source) { // Your File is converted to a ReadableByteChannel from another constructor 
    this(makeReadable(Objects.requireNonNull(source, "source")),
         WHITESPACE_PATTERN);
}

So if your file contains many whitespace characters, you will loop so many times at

while (scan.hasNext()) {
    String next = scan.next();
    rawData.add(next);
}

putting in many objects in your ArrayList but not garbage collecting anything (ie. not freeing memory).

Each call to next() returns the next token until a whitespace is found. Either change the separator, increase your memory size, or change your design.

What is your file format?

来源：https://stackoverflow.com/questions/18689264/out-of-memory-using-scanner-to-read-large-file-into-memory