How can I use proto3 with Hadoop/Spark?

Question

I\'ve got several .proto files which rely on syntax = \"proto3\";. I also have a Maven project that is used to build Hadoop/Spark jobs (Hadoop 2.7.1 and Spark 1

Answer 1

Turns out this kinda thing is documented here: https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html

Just need to relocate the protobuffers and the VerifyError goes away:

          <relocations>
            <relocation>
              <pattern>com.google.protobuf</pattern>
              <shadedPattern>shaded.com.google.protobuf</shadedPattern>
            </relocation>
          </relocations>

Answer 2

Same solution as Dranxo's but with sbt assembly

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.protobuf.*" -> "shadedproto.@1").inProject
    .inLibrary("com.google.protobuf" % "protobuf-java" % protobufVersion)
)