Read large CSV in java

Question

I want to read huge data from CSV, containing around 500,000 rows. I am using OpenCSV library for it. My code for it is like this

    CsvToBean<User> csvConvertor = new CsvToBean<User>();
    List<User> list = null;
    try {
        list =csvConvertor.parse(strategy, new BufferedReader(new FileReader(filepath)));
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }

Upto 200,000 records,data is read into list of User bean objects. But for data more than that I am getting

java.lang.OutOfMemoryError: Java heap space

I have this memory setting in "eclipse.ini" file

-Xms256m
-Xmx1024m

I am thinking a solution of splitting the huge file in separate files and read those files again, which I think is a lengthy solution.

Is there any other way, by which I can avoid OutOfMemoryError exception.

Answer 1

Read line by line

something like this

    CSVReader reader = new CSVReader(new FileReader("yourfile.csv"));
    String [] nextLine;
    while ((nextLine = reader.readNext()) != null) {
        // nextLine[] is an array of values from the line
        System.out.println(nextLine[0] + nextLine[1] + "etc...");
    }

Answer 2

You must set -Xmx value for your app, not eclipse in this case. In "Run configurations", select your app, then go to "Arguments" tab and in the "VM arguments" set that value, for example -Xmx1024m. You can open Run configurations by right clicking in the file you wish to run, then select Run As and then selecting "Run configurations..."

Answer 3

Below Example through you read n number of records from csv file.

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;

public class ReadCSV 
{
    public static void main(String[] args) 
    {
        String csvFile = "C:/Users/LENOVO/Downloads/Compressed/GeoIPCountryWhois.csv";
        BufferedReader br = null;
        String line = "";
        String cvsSplitBy = ",";

        try 
        {
            br = new BufferedReader(new FileReader(csvFile));
            while ((line = br.readLine()) != null) 
            {
                // use comma as separator
                String[] country = line.split(cvsSplitBy);

                System.out.println("Country [code= " + country[4] + " , name=" + country[5] + "]");
            }

        }
        catch (FileNotFoundException e) 
        {
            e.printStackTrace();
        } 
        catch (IOException e) 
        {
            e.printStackTrace();
        } 
        finally 
        {
            if (br != null) 
            {
                try 
                {
                    br.close();
                } 
                catch (IOException e) 
                {
                    e.printStackTrace();
                }
            }
        }
        System.out.println("Done");
    }
}