AssertionError: assertion failed: copyAndReset must return a zero value copy

问题

When I applied ParDo.of(new ParDoFn()) to PCollection named textInput, The program throws this Exception. But The Program is normal when I delete .apply(ParDo.of(new ParDoFn())).

//SparkRunner

private static void testHadoop(Pipeline pipeline){
    Class<? extends FileInputFormat<LongWritable, Text>> inputFormatClass =
            (Class<? extends FileInputFormat<LongWritable, Text>>)
                    (Class<?>) TextInputFormat.class;
    @SuppressWarnings("unchecked")  //hdfs://localhost:9000
            HadoopIO.Read.Bound<LongWritable, Text> readPTransfom_1 = HadoopIO.Read.from("hdfs://localhost:9000/tmp/kinglear.txt",
            inputFormatClass,
            LongWritable.class,
            Text.class);
    PCollection<KV<LongWritable, Text>> textInput = pipeline.apply(readPTransfom_1)
            .setCoder(KvCoder.of(WritableCoder.of(LongWritable.class), WritableCoder.of(Text.class)));

    //OutputFormat
    @SuppressWarnings("unchecked")
    Class<? extends FileOutputFormat<LongWritable, Text>> outputFormatClass =
            (Class<? extends FileOutputFormat<LongWritable, Text>>)
                    (Class<?>) TemplatedTextOutputFormat.class;

    @SuppressWarnings("unchecked")
    HadoopIO.Write.Bound<LongWritable, Text> writePTransform = HadoopIO.Write.to("hdfs://localhost:9000/tmp/output", outputFormatClass, LongWritable.class, Text.class);

    textInput.apply(ParDo.of(new ParDoFn())).apply(writePTransform.withoutSharding());

    pipeline.run().waitUntilFinish();

}

回答1:

Which Spark version are you running on top ? From my experience the error you're getting is thrown by Spark 2.x AccumulatorV2, Spark runner currently supports Spark 1.6.

回答2:

I was facing similar issue when I created custom accumulator that extends org.apache.spark.util.AccumulatorV2. The cause was improper logic in override def isZero: Boolean method. So basically when you copyAndReset method called by default it calls copy() then reset() your isZero() should return true. If you look at the AccumulatorV2 source that is where is a check is:

// Called by Java when serializing an object
final protected def writeReplace(): Any = {
 if (atDriverSide) {
   if (!isRegistered) {
     throw new UnsupportedOperationException(
       "Accumulator must be registered before send to executor")
   }
   val copyAcc = copyAndReset()
   assert(copyAcc.isZero, "copyAndReset must return a zero value copy")
   copyAcc.metadata = metadata
   copyAcc
 } else {
   this
 }
}

specifically this part

 val copyAcc = copyAndReset()
 assert(copyAcc.isZero, "copyAndReset must return a zero value copy")

Hope it helps. Happy sparking!

来源：https://stackoverflow.com/questions/42336251/assertionerror-assertion-failed-copyandreset-must-return-a-zero-value-copy

标签

apache-spark

apache-beam