Illegal quoting in line 1 using Ruby CSV

Question

I am getting this error:

Illegal quoting in line 1. (CSV::MalformedCSVError)

Line 1 in my file is as follows:

\"Status\"    \"Intern         


        

Answer 1

Binary encoding of my file is below:

"\xFF\xFES\x00t\x00a\x00t\x00u\x00s\x00...

0xFF 0xFE is the byte order mark for UTF-16LE.

You have to specify the encoding when processing this file with CSV#foreach:

This method also understands an additional :encoding parameter that you can use to specify the Encoding of the data in the file to be read. You must provide this unless your data is in Encoding::default_external(). CSV will use this to determine how to parse the data. You may provide a second Encoding to have the data transcoded as it is read. For example, encoding: "UTF-32BE:UTF-8" would read UTF-32BE data from the file but transcode it to UTF-8 before CSV parses it.

Furthermore you have to specify that a BOM is present. According to the IO#new docs:

If “BOM|UTF-8”, “BOM|UTF-16LE” or “BOM|UTF16-BE” are (...) present, the BOM is stripped

Applied to your file and example:

CSV.foreach(file, col_sep: "\t", encoding: "BOM|UTF-16LE:UTF-8", headers: true) do |row|
  # ...
end