问题
i have streaming data from few topics kafka, and i want to save each line of RDD into particular table cassandra, my RDD is the collection of case class named Stock :
Stock(test1,2017/07/23 00:01:02,14,Status)
Stock(test1,2017/07/23 00:01:03,78,Status)
Stock(test2,2017/07/23 00:01:02,86,Status)
Stock(test2,2017/07/23 00:01:03,69,Status)
Stock(test3,2017/07/23 00:01:02,46,Status)
Stock(test3,2017/07/23 00:01:03,20,Status)
i want to get the first element of each line in this RDD which represent the name of my topic and give it as name of table in saveToCassandra
any ideas please this is my code without what i want
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
.map(_._2)
messages.foreachRDD(rdd => {
val stockParsed = rdd.map(line => line.split(','))
.map(s => new Stock(s(0).toString, s(1).toString, s(2).toString, s(3).toString))
//here i want to give the fisrt element as name of table
// if i have Stock(test1,2017/07/23 00:01:02,14,Status) table = test1
// if i have Stock(test2,2017/07/23 00:01:02,14,Status) table = test2
.saveToCassandra("sparkcassandra", table, SomeColumns("topic_name", "date_time", "mesure", "status"))
})
ssc.start()
ssc.awaitTermination()
}
}
来源:https://stackoverflow.com/questions/45260052/saving-stream-data-into-tables-cassandra-with-some-names-of-topics