How can i change charset encoding in HTTP response in Java

Question

I have to fetch some JSON object from a remote server and for that i am using this function which is working great except that for sometime some weird data is getting fetched which i believe is because it is using ASCII charset to decode.

Please find below thw method that i am using

public HttpResponse call(String serviceURL,String serviceHost,String namespace,String methodName,String payloadKey, String payloadValue) throws ClientProtocolException,IOException,JSONException
    {
            HttpResponse response = null;
            HttpContext HTTP_CONTEXT = new BasicHttpContext();
            HTTP_CONTEXT.setAttribute(CoreProtocolPNames.USER_AGENT, "Mozilla/5.0");
            HttpPost httppost = new HttpPost(serviceURL);
            httppost.setHeader("User-Agent",Constants.USER_AGENT_BROWSER_FIREFOX);
            httppost.setHeader("Accept", "application/json, text/javascript, */*");
            httppost.setHeader("Accept-Language","en-US,en;q=0.8");
            httppost.setHeader("Content-Encoding", "foo-1.0");
            httppost.setHeader("Content-Type", "application/json; charset=UTF-8");
            httppost.setHeader("X-Requested-With","XMLHttpRequest");
            httppost.setHeader("Host",serviceHost);
            httppost.setHeader("X-Foo-Target", String.format("%s.%s", namespace,methodName));
            /*Making Payload*/
            JSONObject objectForPayload = new JSONObject();
            objectForPayload.put(payloadKey, payloadValue);
            StringEntity stringentity = new StringEntity(objectForPayload.toString());
            httppost.setEntity(stringentity);
            response = client.execute(httppost);
            return response;


    }

All these headers that i am passing are correct and i have verified the same via inspect element in Google chrome or Firebug plugin if you are familiar with Mozilla.

Now the problem is that most of the time i am getting the readable data but sometimes i do get unreadable data.

I debugged using eclipse and noticed that the charset under wrappedEntity is showing as "US-ASCII". I am attaching a jpg for reference

Can someone please tell me how can i change the charset from ASCII to UTF-8 of the response before i do response = client.execute(httppost); . PS:As you have noticed that i am passing charset=utf-8 in the header and that i have already verified using firebug and google chrome that i am passing the exact headers .

Please zoom in to see the image more clearly

Thanks in advance

Answer 1

i was able to resolve the issue just mentioning it for people that may face similar issue. after getting the response first get the entity by using HttpEntity entity = response.getEntity(); and since my response was a json object convert entity to string but using "UTF-8" something like this responseJsonObject = new JSONObject(EntityUtils.toString(entity,"UTF-8"));

previously i was just doing responseJsonObject = new JSONObject(EntityUtils.toString(entity));

Answer 2

You may need to add an "Accept-Encoding"-header and set this to "UTF-8"

Answer 3

I don't think it's a problem with your headers, I think it's a problem with your string. Just having the header say it's utf-8 doesn't mean the string you write is utf-8, and that depends a lot on how the string was encoded and what's in the "payloadValue"

That said, you can always re-encode the thing correctly before sending it across the wire, for example:

objectForPayload.put(payloadKey, payloadValue);
StringEntity stringentity = new StringEntity(
   new String(
      objectForPayload.toString().getBytes(),
      "UTF8"));

See if that works for you.

Answer 4

Just for the record: the "Content-Encoding" header field is incorrect - a correct server would reject the request as it contains an undefined content coding format.

Furthermore, attaching a charset parameter to application/json is meaningless.