AssertionError: assertion failed: copyAndReset must return a zero value copy

Question

When I applied ParDo.of(new ParDoFn()) to PCollection named textInput, The program throws this Exception. But The Program is normal when I delete .apply(ParDo.of(new ParDoFn())).

//SparkRunner

private static void testHadoop(Pipeline pipeline){
    Class<? extends FileInputFormat<LongWritable, Text>> inputFormatClass =
            (Class<? extends FileInputFormat<LongWritable, Text>>)
                    (Class<?>) TextInputFormat.class;
    @SuppressWarnings("unchecked")  //hdfs://localhost:9000
            HadoopIO.Read.Bound<LongWritable, Text> readPTransfom_1 = HadoopIO.Read.from("hdfs://localhost:9000/tmp/kinglear.txt",
            inputFormatClass,
            LongWritable.class,
            Text.class);
    PCollection<KV<LongWritable, Text>> textInput = pipeline.apply(readPTransfom_1)
            .setCoder(KvCoder.of(WritableCoder.of(LongWritable.class), WritableCoder.of(Text.class)));

    //OutputFormat
    @SuppressWarnings("unchecked")
    Class<? extends FileOutputFormat<LongWritable, Text>> outputFormatClass =
            (Class<? extends FileOutputFormat<LongWritable, Text>>)
                    (Class<?>) TemplatedTextOutputFormat.class;

    @SuppressWarnings("unchecked")
    HadoopIO.Write.Bound<LongWritable, Text> writePTransform = HadoopIO.Write.to("hdfs://localhost:9000/tmp/output", outputFormatClass, LongWritable.class, Text.class);

    textInput.apply(ParDo.of(new ParDoFn())).apply(writePTransform.withoutSharding());

    pipeline.run().waitUntilFinish();

}

Answer 1

Which Spark version are you running on top ? From my experience the error you're getting is thrown by Spark 2.x AccumulatorV2, Spark runner currently supports Spark 1.6.

Answer 2

I was facing similar issue when I created custom accumulator that extends org.apache.spark.util.AccumulatorV2. The cause was improper logic in override def isZero: Boolean method. So basically when you copyAndReset method called by default it calls copy() then reset() your isZero() should return true. If you look at the AccumulatorV2 source that is where is a check is:

// Called by Java when serializing an object
final protected def writeReplace(): Any = {
 if (atDriverSide) {
   if (!isRegistered) {
     throw new UnsupportedOperationException(
       "Accumulator must be registered before send to executor")
   }
   val copyAcc = copyAndReset()
   assert(copyAcc.isZero, "copyAndReset must return a zero value copy")
   copyAcc.metadata = metadata
   copyAcc
 } else {
   this
 }
}

specifically this part

 val copyAcc = copyAndReset()
 assert(copyAcc.isZero, "copyAndReset must return a zero value copy")

Hope it helps. Happy sparking!