Apache Spark Kinesis Sample not working

前端 未结 2 1766
慢半拍i
慢半拍i 2020-12-18 16:29

I am trying to run the JavaKinesisWordCountASL example.

The example seem to connect to my Kinesis Stream and gets data from the stream (as shown in the log below).

2条回答
  •  星月不相逢
    2020-12-18 16:46

    Thanks to the hint from @user3594557.

    There are two big notes from https://spark.apache.org/docs/1.1.0/streaming-programming-guide.html#input-dstreams

    If the number of cores allocated to the application is less than or equal to the number of input DStreams / receivers, then the system will receive data, but not be able to process them.

    When running locally, if you master URL is set to “local”, then there is only one core to run tasks. That is insufficient for programs with even one input DStream (file streams are okay) as the receiver will occupy that core and there will be no core left to process the data.

提交回复
热议问题