Fast CSV parsing

Question

I have a java server app that download CSV file and parse it. The parsing can take from 5 to 45 minutes, and happens each hour.This method is a bottleneck of the app so it\'

Answer 1

Quirk-CSV

---

The new kid on the block. It uses java annotations and is built on apache-csv which one of the faster libraries out there for csv parsing.

This library is also thread safe as well if you wanted to re-use the CSVProcessor you can and should.

Example:

Pojo

@CSVReadComponent(type = CSVType.NAMED)
@CSVWriteComponent(type = CSVType.ORDER)
public class Pojo {
    @CSVWriteBinding(order = 0)
    private String name;

    @CSVWriteBinding(order = 1)
    @CSVReadBinding(header = "age")
    private Integer age;

    @CSVWriteBinding(order = 2)
    @CSVReadBinding(header = "money")
    private Double money;

    @CSVReadBinding(header = "name")
    public void setA(String name) {
        this.name = name;
    }

    @Override
    public String toString() {

    return "Name: " + name + System.lineSeparator() + "\tAge: " + age + System.lineSeparator() + "\tMoney: "
            + money;
}}

Main

import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.*;


public class SimpleMain {
public static void main(String[] args) {
    String csv = "name,age,money" + System.lineSeparator() + "Michael Williams,34,39332.15";

    CSVProcessor processor = new CSVProcessor(Pojo.class);
    List list = new ArrayList<>();
    try {
        list.addAll(processor.parse(new StringReader(csv)));
        list.forEach(System.out::println);

        System.out.println();

        StringWriter sw = new StringWriter();
        processor.write(list, sw);
        System.out.println(sw.toString());
    } catch (IOException e) {
    }


}}

Since this is built on top of apache-csv you can use the powerful tool CSVFormat. Lets say the delimiter for the csv are pipes (|) instead of commas(,) you could for Example:

CSVFormat csvFormat = CSVFormat.DEFAULT.withDelimiter('|');
List list = processor.parse(new StringReader(csv), csvFormat);

Another benefit are inheritance is also consider.

For other examples on handling reading/writing non-primitive data