Spark streaming + json4s-jackson dependency problems

Question

I am unable to use json4s-Jackson 3.2.11 within my spark 1.4.1 Streaming application.

Thinking that it was the existing dependency within the spark-core project tha

Answer 1

I had the same issue with emr 4.3.0 and spark 1.6 solved it with installing json4s in the bootstrap action by :

1. down load the json4s jar and put it in s3

2. create the following shell script and put it in s3

    #!/bin/bash
    set -e
    wget -S -T 10 -t 5 https://s3.amazonaws.com/your-bucketname/json4s-native_2.10-3.2.4.jar
    mkdir -p /home/hadoop/lib
    mv json4s-native_2.10-3.2.4.jar /home/hadoop/lib/   
   

3. add it as a bootstrap step in the emr launch steps