Can I set the default string encoding on Ruby 1.9?

Question

This might sound minor, but it's been driving me nuts. Since releasing an application to production last Friday on Ruby 1.9, I've been having lots of minor exceptions related to character encodings. Almost all of it is some variation on:

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8

We have an international user base so plenty of names contain umlauts, etc. If I fix the templates to use force_encoding in a bunch of places, it pops up in the flash message helper. Et cetera.

At the moment it looks like I've nailed down all the ones I knew about, by patching ActiveSupport's string concatenation in one place and then by setting # encoding: utf-8 at the top of every one of my source files. But the feeling that I might have to remember to do that for every file of every Ruby project I ever do from now on, forever, just to avoid string assignment problems, does not sit well in my stomach. I read about the -Ku switch but everything seems to warn that it's for backwards compatibility and might go away at any time.

So my question for 1.9-experienced folks: is setting #encoding in every one of my files really necessary? Is there a reasonable way to do this globally? Or, better, a way to set the default encoding on non-literal values of strings that bypass the internal/external defaults?

Thanks in advance for any suggestions.

Answer 1

Don't confuse file encoding with string encoding

The purpose of the #encoding statement at the top of files is to let Ruby know during reading / interpreting your code, and your editor know how to handle any non-ASCII characters while editing / reading the file -- it is only necessary if you have at least one non-ASCII character in the file. e.g. it's necessary in your config/locale files.

To define the encoding in all your files at once, you can use the magic_encoding gem, it can insert uft-8 magic comment to all ruby files in your app.

The error you're getting at runtime Encoding::CompatibilityError is an error which happens when you try to concatenate two Strings with different encoding during program execution, and their encodings are incompatible.

This most likely happens when:

• you are using L10N strings (e.g. UTF-8), and concatenate them to e.g. ASCII string (in your view)

• the user types in a string in a foreign language (e.g. UTF-8), and your view tries to print it out in some view, along with some fixed string which you pre-defined (ASCII). force_encoding will help there. There's also Encoding::primary_encoding in Rails 1.9 to set the default encoding for new Strings. And there is config.encoding in Rails in the config/application.rb file.

• String which come from your database, and then are combined with other Strings in your view. (their encodings could be either way around, and incompatible).

Side-Note: Make sure to specify a default encoding when you create your database!

    create database yourproject  DEFAULT CHARACTER SET utf8;

If you want to use EMOJIs in your strings:

    create database yourproject DEFAULT CHARACTER SET utf8mb4 collate utf8mb4_bin;

and all indexes on string columns which may contain EMOJI need to be 191 characters in length. CHARACTER SET utf8mb4 COLLATE utf8mb4_bin

The reason for this is that normal UTF8 uses up to 3 bytes, whereas EMOJI use 4 bytes storage.

Please check this Yehuda Katz article, which covers this in-depth, and explains it very well: (there is specifically a section 'Incompatible Encodings')

http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/

http://yehudakatz.com/2010/05/17/encodings-unabridged/

and:

http://zargony.com/2009/07/24/ruby-1-9-and-file-encodings

http://graysoftinc.com/character-encodings

Answer 2

In your config/application.rb add

config.encoding = "utf-8"

and above the Application.initialize! line in config/environment.rb, add following two lines:

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

Hope this helps.

Answer 3

http://zargony.com/2009/07/24/ruby-1-9-and-file-encodings

Don't confuse file encoding and string encoding!

Answer 4

String.module_eval "def initialize\nsuper\nputs encoding\nend"
=> nil
irb(main):006:0> String.new
ASCII-8BIT
=> ""

Not sure how implement your strings in your system, but by hooking into the initialize method of the String object, you can set the encoding for any strings you create in the entire application.