Spark streaming 1.6.1 is not working with Kinesis asl 1.6.1 and asl 2.0.0-preview

问题

I am trying to run spark streaming job on EMR with Kinesis. Spark 1.6.1 with Kinesis ASL 1.6.1. Writing a plain sample wordcount example.

        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kinesis-asl_2.10</artifactId>
        <version>1.6.1</version>
    </dependency>


    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-client</artifactId>
        <version>1.6.3</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>amazon-kinesis-producer</artifactId>
        <version>0.10.2</version>
    </dependency>

This throws following exception

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: com/google/protobuf/ProtocolStringList
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.checkAndSubmitNextTask(ShardConsumer.java:157)
    at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer.consumeShard(ShardConsumer.java:126)

Upgrading to 2.0.0-preview

        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kinesis-asl_2.10</artifactId>
        <version>2.0.0-preview</version>
    </dependency>

gives following exception

java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging

at org.apache.spark.streaming.kinesis.KinesisUtils$$anonfun$createStream$1.apply(KinesisUtils.scala:74)

回答1:

It was caused by protobuf-java dependency conflict. Use mvn dependency:tree to find the version of protobuf-java, which is KCL and KPL depend on. And go to spark lib directory, you would find the another version. Please use maven-shade-plugin, and relocate the conflict classes：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <outputFile>
                    ${project.build.directory}/${project.artifactId}-${project.version}-selfcontained.jar
                </outputFile>
                <relocations>
                    <relocation>
                        <pattern>com.google.protobuf</pattern>
                        <shadedPattern>shade.com.google.protobuf</shadedPattern>
                    </relocation>
                    <relocation>
                        <pattern>com.amazonaws</pattern>
                        <shadedPattern>shade.com.amazonaws</shadedPattern>
                    </relocation>
                </relocations>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

来源：https://stackoverflow.com/questions/38187408/spark-streaming-1-6-1-is-not-working-with-kinesis-asl-1-6-1-and-asl-2-0-0-previe