CSV.read Illegal quoting in line x

Question

I am using ruby CSV.read with massive data. From time to time the library encounters poorly formatted lines, for instance:

\"Illegal quoting in line 53657.\"         


        

Answer 1

Apparently this error can also be caused by unprintable BOM characters. This thread suggests using a file mode to force a conversion, which is what finally worked for me.

require 'csv'

CSV.open(@filename, 'r:bom|utf-8') do |csv|
  # do something
end