Apache Spark Kinesis Sample not working

Question

I am trying to run the JavaKinesisWordCountASL example.

The example seem to connect to my Kinesis Stream and gets data from the stream (as shown in the log below).

Answer 1

Thanks to the hint from @user3594557.

There are two big notes from https://spark.apache.org/docs/1.1.0/streaming-programming-guide.html#input-dstreams

If the number of cores allocated to the application is less than or equal to the number of input DStreams / receivers, then the system will receive data, but not be able to process them.

When running locally, if you master URL is set to “local”, then there is only one core to run tasks. That is insufficient for programs with even one input DStream (file streams are okay) as the receiver will occupy that core and there will be no core left to process the data.

Answer 2

This might have something to do with how many worker thread you got. I had the same problem when I ran the app with --master local[2]. I spent numerous hours searching for an answer and found nothing. Just out of curiosity, I changed to --master local[4] and it worked. I do not know the root cause. Maybe somebody more familiar with Spark can enlighten us.

Note: in my case, my Kinesis stream had two shards. So the app created two input streams, one for each shard.