utf-16

I'm sure this is very easy but I'm getting tied in a knot with all these backslashes. I have some data that I'm scraping (politely) from a website. Occasionally a sentence comes to me looking something like this: u00a362 000? you must be joking Which should of course be '£2 000? you must be joking'. A short test in irb deciphered it. ruby-1.9.2-p180 :001 > string = "u00a3" => "u00a3" ruby-1.9.2-p180 :002 > string = "\u00a3" => "£" Of course: add a backslash and it will be decoded. I created the following with the help of this question : puts str.gsub('u00', '\\u00') which resulted in \u00a3

I'm trying to URL-encode some strings, however I have problems with methods provided by the .Net framework. For instance, I'm trying the encode strings that contain the 'â' character. According to w3schools for instance, I would expect this caracter to be encoded as '%E2' (and a PHP system I must call expects this too...). I tried using these methods: System.Web.HttpUtility.UrlEncode("â"); System.Web.HttpUtility.UrlPathEncode("â"); Uri.EscapeUriString("â"); Uri.EscapeDataString("â"); However, they all encode this character as: %C3%A2 I suppose this has something to do with the fact that

With this code: test.py import sys import codecs sys.stdout = codecs.getwriter('utf-16')(sys.stdout) print "test1" print "test2" Then I run it as: test.py > test.txt In Python 2.6 on Windows 2000, I'm finding that the newline characters are being output as the byte sequence \x0D\x0A\x00 which of course is wrong for UTF-16. Am I missing something, or is this a bug? Try this: import sys import codecs if sys.platform == "win32": import os, msvcrt msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY) class CRLFWrapper(object): def __init__(self, output): self.output = output def write(self, s): self

I need to convert utf16 text to utf8. The actual conversion code is simple: std::wstring in(...); std::string out = boost::locale::conv::utf_to_utf<char, wchar_t>(in); However the issue is that the UTF16 is read from a file and it may or may not contain BOM. My code needs to be portable (minimum is windows/osx/linux). I'm really struggling to figure out how to create a wstring from the byte sequence. EDIT: this is not a duplicate of the linked question, as in that question the OP needs to convert a wide string into an array of bytes - and I need to convert the other way around. You should not

I am working on this code which receives a cpprest sdk response containing a base64_encoded payload which is a json. here is my code snippet: typedef std::wstring string_t; //defined in basic_types.h in cpprest lib void demo() { http_response response; //code to handle respose ... json::value output= response.extract_json(); string_t payload = output.at(L"payload").as_string(); vector<unsigned char> base64_encoded_payload = conversions::from_base64(payload); std::string utf8_payload(base64_encoded_payload.begin(), base64_encoded_payload.end()); //in debugger I see the Japanese chars are

On MSVC converting utf-16 to utf-32 is easy - with C11's codecvt_utf16 locale facet. But in GCC (gcc (Debian 4.7.2-5) 4.7.2) seemingly this new feature hasn't been implemented yet. Is there a way to perform such conversion on Linux without iconv (preferrably using conversion tools of std library)? Decoding UTF-16 into UTF-32 is extremely easy. You may want to detect at compile time the libc version you're using, and deploy your conversion routine if you detect a broken libc (without the functions you need). Inputs: a pointer to the source UTF-16 data ( char16_t * , ushort * , -- for

I have UTF-16 hex representation such as “0633064406270645” which is "سلام" in Arabic language. I would like to convert it to its text equivalent. Is there a straight way to do that in PostgreSQL? I can convert the UTF code point like below; unfortunately it seems UTF16 is not supported. Any ideas on how to do it in PostgreSQL, worst case I will write a function? SELECT convert_from (decode (E'D8B3D984D8A7D985', 'hex'),'UTF8'); "سلام" SELECT convert_from (decode (E'0633064406270645', 'hex'),'UTF16'); ERROR: invalid source encoding name "UTF16" ********** Error ********** That's right, Postgres