Spark Direct Streaming - consume same message in multiple consumers

问题

How to consume Kakfa topic messages in multiple Consumers using Direct Stream approach?

Is it possible? Since Direct Stream approach doesn't have Consumer Group concept.

What happens, if i pass group.id as kafkaparams for the DirectStream method? The below code works with group.id as Kafka Params also without group.id.

Sample Code:

val kafkaParams = Map(
  "group.id" -> "group1", 
  CommonClientConfigs.SECURITY_PROTOCOL_CONFIG -> sasl,
  ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
  ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer",
  "metadata.broker.list" -> brokerList,
  "zookeeper.connect" -> zookeeperURL
)

val dStream = 
  KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](
    ssc, kafkaParams, topicSet
  ).map(_._2)

回答1:

Nothing happens, Sparks direct streaming approach doesn't take into account the group id parameter at all as it uses the lower level SimpleConsumer. You can't consume the same topic with different Spark direct approach streams. You can defer to the older receiver based approach which does utilize groups.

来源：https://stackoverflow.com/questions/42155542/spark-direct-streaming-consume-same-message-in-multiple-consumers