Task not serializable exception while running apache spark job

Question

The following java program was written to experiment with apache spark.

The program tries to read a list of positive and negative words from a respective file, compa

Answer 1

When you create an anonymous class, the compiler does some stuff:

JavaRDD numAs = positiveComments.filter(new Function()
      {
        public Boolean call(String s)
        {
          return s.contains(iterator.next());
        }
      });

It will be rewritten as:

JavaRDD numAs = positiveComments.filter(new Function()
      {
        private Iterator<...> $iterator;
        public Boolean call(String s)
        {
          return s.contains($iterator.next());
        }
      });

This is why you can have a NotSerializableException because the Iterator is not serializable.

To avoid that, simply extract the result of next before:

String value = iterator.next();
JavaRDD numAs = positiveComments.filter(new Function()
      {
        public Boolean call(String s)
        {
          return s.contains(value);
        }
      });