Is it possible to turn off quote processing in the Postgres COPY command with CSV format?

Question

I have CSV files, tab-separated, fields not wrapped in quotes, where field data can contain characters like single quotes, double quotes, pipes and backslashes.

Answer 1

(Added as a new answer since I don't have the reputation yet to comment.)

For the record, since I've been struggling with the same issue, you can use tr to remove \b, instead of just hoping it's not in your text anywhere.

tr -d '\010' < filename.csv > newfile.csv

(Using that \010 is the octal representation of \b).

Since COPY supports reading from STDIN, you can ease the I/O impact by piping tr's output:

cat filename.csv | tr -d '\010' | COPY <tablename> FROM STDIN WITH CSV DELIMITER E'\t' QUOTE E'\b' NULL AS '';

Answer 2

Workaround (thanks to this comment!)

COPY <tablename> FROM <filename> WITH CSV DELIMITER E'\t' QUOTE E'\b' NULL AS '';

So basically specifying a quote character that should never be in the text, but that's pretty ugly.

I'd much prefer it if there was in fact a way to turn off quote processing altogether.

Answer 3

The mode you want to use for data formatted as you describe is the default text mode. It will pass most characters unhindered into the database. It does not have quote processing, and it's using tabs as delimiters. Using CSV mode will just cause you trouble because you're introducing quoting that you have to work around.

Text mode will pass dollar characters, single and double quotes, pipes, and even backspaces (even though that was not mentioned in the question) right in. The one thing in the example that's not passed through is backslashes. But that's as simple as escaping them, for example by this sed command:

sed -e 's/\\/\\\\/g' < source.txt > processed.txt

Then the processed file should be importable without any additional options:

\copy sometable from processed.txt