utf-16

For some reason i can not seek my utf16 file. It produces 'UnicodeException: UTF-16 stream does not start with BOM'. My code: f = codecs.open(ai_file, 'r', 'utf-16') seek = self.ai_map[self._cbClass.Text] #seek is valid int f.seek(seek) while True: ln = f.readline().strip() I tried random stuff like first reading something from stream, didnt help. I checked offset that is seeked to using hex editor - string starts at character, not null byte (i guess its good sign, right?) So how to seek utf-16 in python? Well, the error message is telling you why: it's not reading a byte order mark. The byte

According to this , SQL Server 2K5 uses UCS-2 internally. It can store UTF-16 data in UCS-2 (with appropriate data types, nchar etc), however if there is a supplementary character this is stored as 2 UCS-2 characters. This brings the obvious issues with the string functions, namely that what is one character is treated as 2 by SQL Server. I am somewhat surprised that SQL Server is basically only able to handle UCS-2, and even more so that this is not fixed in SQL 2K8. I do appreciate that some of these characters may not be all that common. Aside from the functions suggested in the article,

(In .NET) I have arbitrary binary data stored in in a byte[] (an image, for example). Now, I need to store that data in a string (a "Comment" field of a legacy API). Is there a standard technique for packing this binary data into a string ? By "packing" I mean that for any reasonably large and random data set, bytes.Length/2 is about the same as packed.Length ; because two bytes are more-or-less a single character. The two "obvious" answers don't meet all the criteria: string base64 = System.Convert.ToBase64String(bytes) doesn't make very efficient use of the string since it only uses 64

Is there any way to make git gui display and show diffs for UTF16 files somehow? I found some information , but this is mostly referring to the command line rather than the gui. I have been working on a much better solution with help from the msysGit people, and have come up with this clean/smudge filter. The filter uses the Gnu file and iconv commands to determine the type of the file, and convert it to and from msysGit's internal UTF-8 format. This type of Clean/Smudge Filter gives you much more flexibility. It should allow Git to treat your mixed-format files as UTF-8 text in most cases:

I'm using this code to generate U+10FFFC var s = Encoding.UTF8.GetString(new byte[] {0xF4,0x8F,0xBF,0xBC}); I know it's for private-use and such, but it does display a single character as I'd expect when displaying it. The problems come when manipulating this unicode character. If I later do this: foreach(var ch in s) { Console.WriteLine(ch); } Instead of it printing just the single character, it prints two characters (i.e. the string is apparently composed of two characters). If I alter my loop to add these characters back to an empty string like so: string tmp=""; foreach(var ch in s) {

What's the easiest way to convert XML from UTF16 to a UTF8 encoded file? Ben Laan This may not be the most optimal, but it works. Simply load the xml and push it back out to a file. the xml heading is lost though, so this has to be re-added. $files = get-ChildItem "*.xml" foreach ( $file in $files ) { [System.Xml.XmlDocument]$doc = new-object System.Xml.XmlDocument; $doc.set_PreserveWhiteSpace( $true ); $doc.Load( $file ); $root = $doc.get_DocumentElement(); $xml = $root.get_outerXml(); $xml = '<?xml version="1.0" encoding="utf-8"?>' + $xml $newFile = $file.Name + ".new" Set-Content -Encoding