Encoding::UndefinedConversionError

Question

I keep getting an Encoding::UndefinedConversionError - "\\xC2" from ASCII-8BIT to UTF-8 every time I try to convert a hash into a JSON string. I tried

Answer 1

menu.to_s.encode('UTF-8', invalid: :replace, undef: :replace, replace: '?')

This worked perfectly, I had to replace some extra characters but there are no more errors.

Answer 2

What do you expect for "\xC2"? Probably a Â

With ASCII-8BIT you have binary data, and ruby cant decide, what should be.

You must first set the encoding with force_encoding.

You may try the following code:

Encoding.list.each{|enc|
  begin
    print "%-10s\t" % [enc]
    print "\t\xC2".force_encoding(enc)
    print "\t\xC2".force_encoding(enc).encode('utf-8')
  rescue => err
    print "\t#{err}"
  end
  print "\n"
}

The result are the possible values in different encodings for your "\xC2".

The result may depend on your Output format, but I think you can make a good guess, which encoding you have.

When you defined the encoding you need (probably cp1251) you can

menu.force_encoding('cp1252').to_json

See also Kashyaps comment.

Answer 3

If you don't care about losing the strange characters, you can blow them away:

str.force_encoding("ASCII-8BIT").encode('UTF-8', undef: :replace, replace: '')

Answer 4

:fallback option can be useful if you know what chars you want to replace

"Text                                                                     

                                                        
            

Answer 5

Your auto-accepted solution doesn't work, there are effectively no errors, but it is NOT JSON.

I solved the problem using the oj gem, it now works find. It is also faster than the standard JSON library.

Writting :

   menu_json = Oj.dump menu

Reading :

   menu2 = Oj.load menu_json

https://github.com/ohler55/oj for more details. I hope it will help.