How to guess the encoding of a file with no BOM in .NET?

Question

I\'m using the StreamReader class in .NET like this:

using( StreamReader reader = new StreamReader( \"c:\\somefile.html\", true ) {
    string filetext = rea         


        

Answer 1

I had good luck with Pude, a C# port of Mozilla Universal Charset Detector.

Answer 2

A hacky technique might be to take an MD5 of the text, then decode the text and re-encode it in various encodings, MD5'ing each one. If one matches you guess it's that encoding.

That's obviously too slow for something that handles a lot of files but for something like a text editor I could see it working.

Other than that, it'll be hands dirty porting the java libraries from this post that came from the Delphi SO question, or using the IE MLang feature.

Answer 3

UTF-8 is designed in a way that it is unlikely to have a text encoded in an arbitrary 8bit-encoding like latin1 being decoded to proper unicode using UTF-8.

So the minimum approach is this (pseudocode, I don't talk .NET):

try: u = some_text.decode("UTF-8") except UnicodeDecodeError: u = some_text.decode("most-likely-encoding")

For the most-likely-encoding one usually uses e.g. latin1 or cp1252 or whatever. More sophisticated approaches might try & find language-specific character pairings, but I'm not aware of something that does that as a library or some such.

Answer 4

I used this to do something similar a while back:

http://www.conceptdevelopment.net/Localization/NCharDet/

Answer 5

See my (recent) answer to this (as far as I can tell, equivalent) question: How can I detect the encoding/codepage of a text file

It does NOT attempt to guess across a range of possible "national" encodings like MLang and NCharDet do, but rather assumes you know what kind of non-unicode files you're likely to encounter. As far as I can tell from your question, it should address your problem pretty reliably (without relying on the "black box" of MLang).

Answer 6

You should read this article by Raymond Chen. He goes into detail on how programs can guess what an encoding is (and some of the fun that comes from guessing).

Some files come up strange in Notepad