Retrofit - removing some invalid characters from response body before parsing it as json

Question

I have an external web service that in the response body returns json but nested in parentheses, like this:

({\"door_x\":\"103994.001461\",\"door_y\":\"98780         


        

Answer 1

Alternative regex to convert from jsonp (dirty) to json (clean):

String clean = dirty.replaceFirst("(?s)^\\((.*)\\)$", "$1");

Answer 2

Orginally answered here

For parsing invalid JSON or String or JSONP response, use ScalarConverterFactory.

For parsing JSON response, use GsonConverterFactory.

If you use say flatMap to call JSON API followed by JSONP API then use both GsonConverterFactory(needed for JSON) and ScalarConverterFactory(needed for JSONP).

Make sure you have below dependencies in your gradle :

implementation 'com.squareup.retrofit2:converter-gson:2.5.0'
//For serialising JSONP add converter-scalars
implementation 'com.squareup.retrofit2:converter-scalars:2.1.0'
//An Adapter for adapting RxJava 2.x types.
implementation 'com.squareup.retrofit2:adapter-rxjava2:2.2.0'

Add converterFactories to retrofit and use setLenient() while building Gson to get rid of error JSON document was not fully consumed.

val gson = GsonBuilder()
            .setLenient()
            .create()

val retrofit = Retrofit.Builder()
            .baseUrl("http://api.flickr.com/")
            .client(builder.build())
            .addConverterFactory(ScalarsConverterFactory.create()) //important
            .addConverterFactory(GsonConverterFactory.create(gson))
            .addCallAdapterFactory(RxJava2CallAdapterFactory.create())
            .build()

@GET("end-point/to/some/jsonp/url")
fun getJsonpData() : Observable<String>

Use converters to convert invalid JSON to JSON by removing prefix and suffix present. And then convert the string to your data model via

SomeDataModel model = Gson().fromJson<SomeDataModel>(jsonResponse,
            SomeDataModel::class.java)

Answer 3

Solution For Retrofit 2

The code below is the same as the GsonConverter except you can edit the Response before converting to its model

Edit public T convert(ResponseBody value) to clean your Response

/**
 * Modified by TarekkMA on 8/2/2016.
 */

public class MyJsonConverter extends Converter.Factory {

    public static MyJsonConverter create() {
        return create(new Gson());
    }

    public static JsonHandler create(Gson gson) {
        return new JsonHandler(gson);
    }

    private final Gson gson;

    private JsonHandler(Gson gson) {
        if (gson == null) throw new NullPointerException("gson == null");
        this.gson = gson;
    }

    @Override
    public Converter<ResponseBody, ?> responseBodyConverter(Type type, Annotation[] annotations,
                                                            Retrofit retrofit) {
        TypeAdapter<?> adapter = gson.getAdapter(TypeToken.get(type));
        return new GsonResponseBodyConverter<>(gson, adapter);
    }

    @Override
    public Converter<?, RequestBody> requestBodyConverter(Type type,
                                                          Annotation[] parameterAnnotations, Annotation[] methodAnnotations, Retrofit retrofit) {
        TypeAdapter<?> adapter = gson.getAdapter(TypeToken.get(type));
        return new GsonRequestBodyConverter<>(gson, adapter);
    }


    final class GsonRequestBodyConverter<T> implements Converter<T, RequestBody> {
        private final MediaType MEDIA_TYPE = MediaType.parse("application/json; charset=UTF-8");
        private final Charset UTF_8 = Charset.forName("UTF-8");

        private final Gson gson;
        private final TypeAdapter<T> adapter;

        GsonRequestBodyConverter(Gson gson, TypeAdapter<T> adapter) {
            this.gson = gson;
            this.adapter = adapter;
        }

        @Override
        public RequestBody convert(T value) throws IOException {
            Buffer buffer = new Buffer();
            Writer writer = new OutputStreamWriter(buffer.outputStream(), UTF_8);
            JsonWriter jsonWriter = gson.newJsonWriter(writer);
            adapter.write(jsonWriter, value);
            jsonWriter.close();
            return RequestBody.create(MEDIA_TYPE, buffer.readByteString());
        }
    }

    final class GsonResponseBodyConverter<T> implements Converter<ResponseBody, T> {
        private final Gson gson;
        private final TypeAdapter<T> adapter;

        GsonResponseBodyConverter(Gson gson, TypeAdapter<T> adapter) {
            this.gson = gson;
            this.adapter = adapter;
        }

        @Override
        public T convert(ResponseBody value) throws IOException {
            String dirty = value.string();
            String clean = dirty.replace("<?xml version=\"1.0\" encoding=\"utf-8\"?>\r\n" +
                    "<string xmlns=\"http://tempuri.org/\">","").replace("</string>","");
            try {
                return adapter.fromJson(clean);
            } finally {
                value.close();
            }
        }
    }


}

• Another solution is to blame the inexperienced backend developer.

Answer 4

You can clean painlessly the response in your GsonConverter before Gson deserialized the body into type object.

 public class CleanGsonConverter extends GsonConverter{

            public CleanGsonConverter(Gson gson) {
                super(gson);
            }

            public CleanGsonConverter(Gson gson, String encoding) {
                super(gson, encoding);
            }

            @Override
            public Object fromBody(TypedInput body, Type type) throws ConversionException {
                String dirty = toString(body);
                String clean = dirty.replaceAll("(^\\(|\\)$)", "");
                body = new JsonTypedInput(clean.getBytes(Charset.forName(HTTP.UTF_8)));
                return super.fromBody(body, type);
            }
            private String toString(TypedInput body){
                    BufferedReader br = null;
                    StringBuilder sb = new StringBuilder();
                    String line;
                    try {
                        br = new BufferedReader(new InputStreamReader(body.in()));
                        while ((line = br.readLine()) != null) {
                            sb.append(line);
                        }

                    } catch (IOException e) {
                        e.printStackTrace();
                    } finally {
                        if (br != null) {
                            try {
                                br.close();
                            } catch (IOException e) {
                                e.printStackTrace();
                            }
                        }
                    }

                    return sb.toString();

                }
        };

JsonTypedInput:

   public class JsonTypedInput implements TypedInput{

        private final byte[] mStringBytes;

        JsonTypedInput(byte[] stringBytes) {
            this.mStringBytes = stringBytes;
        }


        @Override
        public String mimeType() {
            return "application/json; charset=UTF-8";
        }



        @Override
        public long length() {
            return mStringBytes.length;
        }

        @Override
        public InputStream in() throws IOException {
            return new ByteArrayInputStream(mStringBytes);
        }
    }

Here I subclassed GsonConverter to get access to the response before it is converted to object. JsonTypedOutput is used to preserve the mime type of the response after cleaning it from the junk chars.

Usage:

restAdapterBuilder.setConverter(new CleanGsonConverter(gson));

Blame it on your backend guys. :)