Dealing with kanji characters in C++

Question

I have a windows deskop application (named: Timestamp) written in C++ that use .NET called CLR.

I also have DLL project (named: Amscpprest) written in native c++ and uses CPPREST SDK to get json data from server and pass the data to my Timestamp app.

Here's the scenario: This is the return json data from my server, its a list of staff name and most of it is japanese names written in Kanji characters.

[ 
  {   
     "staff": {
         "id": 121,
         "name": "福士 達哉",
         "department": [
            {
               "_id": 3,
               "name": "事業推進本部"
            }
         ]    
      } 
  },
  {   
     "staff": {
         "id": 12,
         "name": "北島 美奈",
         "department": [
            {
               "_id": 4,
               "name": "事業開発本部"
            }
         ]    
      } 
  },
  {   
     "staff": {
         "id": 151,
         "name": "大河原 紗希",
         "department": [
            {
               "_id": 3,
               "name": "事業推進本部"
            }
         ]    
      } 
  }
]

This my code in my DLL project (Amscpprest). This is how get the data and pass to my CLR project:

std::map<int, std::string> staffMap;
auto GetStaffMap() -> std::map<int, std::string> {
    return staffMap;
}

void display_json(json::value const & jvalue, utility::string_t const & prefix)
{
    try {
        //==== Iterate through json data and make an associative array ====/
        auto DataArray = jvalue.as_array();

        // loop through 'data' object
        for (int i = 0; i < DataArray.size(); i++)
        {
            try
            {
                auto data = DataArray[i];
                auto dataObj = data.at(U("staff")).as_object();

                int key;
                std::string value;

                // loop through each object of 'data'
                for (auto iterInner = dataObj.cbegin(); iterInner != dataObj.cend(); ++iterInner)
                {
                    auto &propertyName = iterInner->first;
                    auto &propertyValue = iterInner->second;

                    if (propertyName == L"_id")
                    {
                        key = propertyValue.as_integer();
                    }
                    else if (propertyName == L"name")
                    {
                        value = conversions::to_utf8string(propertyValue.as_string());
                    }
                }
                staffMap.insert(std::make_pair(key, value));
            }
            catch (const std::exception& e)
            {
                std::wcout << e.what() << std::endl;
            }
        }

    }
    catch (const std::exception& e) {
        std::wcout << e.what() << std::endl;
    }
}

pplx::task<http_response> task_request(http_client & client, method mtd, json::value const & jvalue, std::string searchText)
{
    //std::string  url = "/api/authenticate/searchStaffs/";
    std::string url = "/api/authenticate/oldgms/staffs_id_name/";

    return client.request(mtd, utility::conversions::to_string_t(url));
}

void make_request(http_client & client, method mtd, json::value const & jvalue, std::string searchText)
{
    task_request(client, mtd, jvalue, searchText)
        .then([](http_response response)
    {
        if (response.status_code() == status_codes::OK)
        {
            return response.extract_json();
        }
        return pplx::task_from_result(json::value());
    })
        .then([](pplx::task<json::value> previousTask)
    {
        try
        {
            display_json(previousTask.get(), L"R: ");
        }
        catch (http_exception const & e)
        {
            std::wcout << e.what() << std::endl;
        }
    })
        .wait();
}


int SearchStaff(std::string searchText)
{
    //clear staffMap every call
    staffMap.clear();

    http_client client(U("http://52.68.13.154:3000"));

    auto nullValue = json::value::null();
    //std::string search_text = conversions::to_utf8string(L"北島 美奈");
    make_request(client, methods::GET, nullValue, searchText);

    return staff_id;
}

And this is my code in my CLR project (Timestamp). This is how i accept the data from my dll project and display to the user interface.

String^ input = searchBox->Text;

std::string searchText = msclr::interop::marshal_as<std::string>(input);

// Clear listView item every type in searchbox
listView1->Items->Clear();

Staffs::SearchStaff(searchText);
std::map<int, std::string> staffMap = Staffs::GetStaffMap();

std::map<int, std::string>::iterator iter;
for (iter = staffMap.begin(); iter != staffMap.end(); iter++)
{
    String^ value = msclr::interop::marshal_as<System::String^>(iter->second);
    int key = iter->first;

    listViewItem = gcnew Windows::Forms::ListViewItem(value);
    listViewItem->SubItems->Add(System::Convert::ToString(key));
    this->listView1->Items->Add(this->listViewItem);
}

I expect that it should display the names and id correctly in the listview but this was the result:

I'm hoping that someone could help me solve this problem.

Answer 1

I think you have two distinct issues here.

First, in the iteration, you try to read a key called _id, which isn't there (it should be id), so your int key is never assigned a value (and it's not initialized, that's why you get a weird number in your list view).

Second, you have to convert utf8 (stored in std::string) to ucs2, which is what a .NET String is made of. You can achieve that using the UTF8Encoding class (docs here) . So, instead of this:

String^ value = msclr::interop::marshal_as<System::String^>(iter->second);

you need something like this:

//make a byte array to hold the string chars
array<Byte>^ bytes = gcnew array<Byte>(iter->second.size());

//copy the string chars into the byte array
System::Runtime::InteropServices::Marshal::Copy(IntPtr(&iter->second[0]), bytes, 0, iter->second.size());

//get a string from the bytes, using UTF8Encoding
String^ value = System::Text::UTF8Encoding::UTF8->GetString(bytes);