Avro schema doesn't honor backward compatibilty

Question

I have this avro schema

{
 \"namespace\": \"xx.xxxx.xxxxx.xxxxx\",
 \"type\": \"record\",
 \"name\": \"MyPayLoad\",
 \"fields\": [
     {\"name\": \"filed1\"         


        

Answer 1

There are two possible issues that i can see in your schema

1. The default value for me always seems to have work as null to specify this you need to set

"default": null

2. Also in your schema you did forget to add a , (field separator) between the array and new field. Hence try changing your schema as

{ "namespace": "xx.xxxx.xxxxx.xxxxx", "type": "record", "name": "MyPayLoad", "fields": [ {"name": "filed1", "type": "string"}, {"name": "filed2", "type": "long"}, {"name": "filed3", "type": "boolean"}, { "name" : "metrics", "type": { "type" : "array", "items": { "name": "MyRecord", "type": "record", "fields" : [ {"name": "min", "type": "long"}, {"name": "max", "type": "long"}, {"name": "sum", "type": "long"}, {"name": "count", "type": "long"} ] } } }, {"name": "agentType", "type": ["null", "string"], "default":null} ] }

Answer 2

finally i got this working. I need to give both the schemas in the SpecificDatumReader So i modified the parsing like this where i passed both the old and new schema in the reader and it worked like a charm

public static final MyPayLoad parseBinaryPayload(byte[] payload) {
        DatumReader<MyPayLoad> payloadReader = new SpecificDatumReader<>(SCHEMA_V1, SCHEMA_V2);
        Decoder decoder = DecoderFactory.get().binaryDecoder(payload, null);
        MyPayLoad myPayLoad = null;
        try {
            myPayLoad = payloadReader.read(null, decoder);
        } catch (IOException e) {
            logger.log(Level.SEVERE, e.getMessage(), e);
        }

        return myPayLoad;
    }

Answer 3

I am facing this exact situation. Data written by the older schema fails when trying to read it with the newer schema. The newer schema has just one additional field with union and default set. "type":["null","string"],"doc":"","default":null

In spite of setting default, the null doesn't get filled in automatically during reading. Both the writer and the reader schemas need to be provided during reading. My understanding was avro is backward compatible and it should be able to support newer columns without the need for the older schema.