avro error on AWS EMR

Question

I\'m using spark-redshift (https://github.com/databricks/spark-redshift) which uses avro for transfer.

Reading from Redshift is OK, while writing I\'m getting

Answer 1

Jonathan from EMR here. Part of the problem is that Hadoop depends upon Avro 1.7.4, and the full Hadoop classpath is included in the Spark path on EMR. It might help for us to upgrade Hadoop's Avro dependency to 1.7.7 so that it matches with Spark's Avro dependency, though I'm a little afraid that this might break something else, but I can try it out anyway.

BTW, one problem I noticed with your example EMR cluster config is that you're using the "spark-env" config classification, whereas the "spark-defaults" classification would be the appropriate one for setting spark.{driver,executor}.userClassPathFirst. I'm not sure this by itself would solve your problem though.