avro error on AWS EMR

Question

I\'m using spark-redshift (https://github.com/databricks/spark-redshift) which uses avro for transfer.

Reading from Redshift is OK, while writing I\'m getting

Answer 1

spark-redshift maintainer here.

Other EMR users have encountered similar errors when using newer versions of the spark-avro library (which spark-redshift depends on). In a nutshell, the problem seems to be that EMR's older version of Avro takes precedence over the new version required by spark-avro. At https://github.com/databricks/spark-avro/issues/91, an issue that seems to match the exception reported here, one user suggested embedding the Avro JARs with their application code: https://github.com/databricks/spark-avro/issues/91#issuecomment-142543149