Importing CSV that has line breaks within the actual fields

Question

I am using PHP to import a CSV file, which originates from an excel spreadsheet. Some of the fields contain line breaks in them, so when I reopen the csv in excel / open office spreadsheet, it misinterprets where the line breaks should happen.

Also in my script, using fgetcsv to go through each line, it is incorrectly line breaking where it shouldn't be.

I could manually cleanse the data but a) that would take ages as its a 10k line file, and b) the data is exported from a clients existing piece of software

Any ideas on how to automatically solve this on the import process? I would have thought delimiting the fields would have sorted it but it does not.

Answer 1

I had that problem too and did not find an way to read the data correctly.

In my case it was an one-time-import, so i made an script that searched for all line-breaks within an column and replaced it with something like #####. Then I imported the data and replaced that by linebreaks.

If you need an regular import you could write you own CSV-Parser, that handles the problem. If the text-columns are within "" you could treat everything between two "" as one columns (with check for escaped " within the content).

Answer 2

The accepted answer didn't solve the problem for me, but I eventually found this CSV parser library on google code that works well for multiline fields in CSV's.

parsecsv-for-php:
https://github.com/parsecsv/parsecsv-for-php

---

For historical purposes, the original project home was:
http://code.google.com/p/parsecsv-for-php/

Answer 3

My solution is the following:

nl2br(string);

http://php.net/manual/en/function.nl2br.php

Once you get to the individual cell (string) level, run it on the string and it will convert the linebreaks to html breaks for you.

Answer 4

Yes you needs to find that comma and replace by some special characters like combination of {()} and finally replace them with , that you are originally looking for.

Hope that helps you.

Answer 5

Although it is old question the answer might be still relevant to ppl. There is currently new library (framework independent) http://csv.thephpleague.com/ which supports NL chars in fields as well as some filtering.

Answer 6

It's an old thread but i encountered this problem and i solved it with a regex so you can avoid a library just for that. Here the code is in PHP but it can be adapted to other language.

$parsedCSV = preg_replace('/(,|\n|^)"(?:([^\n"]*)\n([^\n"]*))*"/', '$1"$2 $3"', $parsedCSV);

This solutions supposes the fields containing a linebreak are enclosed by double quotes, which seems to be a valid assumption, at least for what i have seen so far. Also, the double quotes should follow a , or be placed at the start of a new line (or first line).

Example:

field1,"field2-part1\nfield2-part2",field3

Here the \n is replaced by a whitespace so the result would be:

field1,"field2-part1 field2-part2",field3

The regex should handle multiple linebreaks as well.

This might not be efficient if the content is too large, but it can help for many cases and the idea can be reused, maybe optimized by doing this for smaller chunks (but you'd need to handle the cuts with fix-sized buffered).