问题
I'm trying to run a simple pipeline on local cluster using Protocol Buffer to pass data between Beam functions. The com.google.protobuf:protobuf-java is included in FatJar.
Everything works fine if I run it through:
java -jar target/dataflow-test-1.0-SNAPSHOT.jar \
--runner=org.apache.beam.runners.flink.FlinkRunner \
--input=/tmp/kinglear.txt --output=/tmp/wordcounts.txt
But it fails when trying to run on flink cluster:
flink run target/dataflow-test-1.0-SNAPSHOT.jar \
--runner=org.apache.beam.runners.flink.FlinkRunner \
--input=/tmp/kinglear.txt --output=/tmp/wordcounts.txt
Ready to run project: https://github.com/orian/beam-flink-local-cluster
Beam version is 0.3-Snapshot (bleeding edge) and it uses version 1.0.3 of Flink, my local cluster runs version 1.0.3. Flink uses a protobuf-java 2.5.0.
The program finished with the following exception:
java.lang.NoSuchMethodError: com.google.protobuf.ExtensionRegistry.getAllImmutableExtensionsByExtendedType(Ljava/lang/String;)Ljava/util/Set;
at org.apache.beam.sdk.coders.protobuf.ProtobufUtil.recursivelyAddDescriptors(ProtobufUtil.java:129)
at org.apache.beam.sdk.coders.protobuf.ProtobufUtil.getRecursiveDescriptorsForClass(ProtobufUtil.java:63)
at org.apache.beam.sdk.coders.protobuf.ProtobufUtil.verifyDeterministic(ProtobufUtil.java:92)
at org.apache.beam.sdk.coders.protobuf.ProtoCoder.verifyDeterministic(ProtoCoder.java:258)
at org.apache.beam.sdk.transforms.GroupByKey.validate(GroupByKey.java:196)
at org.apache.beam.sdk.transforms.GroupByKey.validate(GroupByKey.java:121)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at org.apache.beam.sdk.transforms.Combine$PerKey.apply(Combine.java:1851)
at org.apache.beam.sdk.transforms.Combine$PerKey.apply(Combine.java:1740)
at org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at org.apache.beam.runners.flink.FlinkRunner.apply(FlinkRunner.java:196)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at org.apache.beam.sdk.transforms.Count$PerElement.apply(Count.java:115)
at org.apache.beam.sdk.transforms.Count$PerElement.apply(Count.java:100)
at org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at org.apache.beam.runners.flink.FlinkRunner.apply(FlinkRunner.java:196)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at com.mycompany.dataflow.WordCount$CountWords.apply(WordCount.java:74)
at com.mycompany.dataflow.WordCount$CountWords.apply(WordCount.java:63)
at org.apache.beam.sdk.runners.PipelineRunner.apply(PipelineRunner.java:76)
at org.apache.beam.runners.flink.FlinkRunner.apply(FlinkRunner.java:196)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:400)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at com.mycompany.dataflow.WordCount.main(WordCount.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1192)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243)
来源：https://stackoverflow.com/questions/39288943/apache-beam-with-flink-backend-throws-nosuchmethoderror-on-calls-to-protobuf-jav