Saving stream data into tables cassandra with some names of topics

◇◆丶佛笑我妖孽 提交于 2019-12-11 06:05:46

问题


i have streaming data from few topics kafka, and i want to save each line of RDD into particular table cassandra, my RDD is the collection of case class named Stock :

Stock(test1,2017/07/23 00:01:02,14,Status)
Stock(test1,2017/07/23 00:01:03,78,Status)
Stock(test2,2017/07/23 00:01:02,86,Status)
Stock(test2,2017/07/23 00:01:03,69,Status)
Stock(test3,2017/07/23 00:01:02,46,Status)
Stock(test3,2017/07/23 00:01:03,20,Status)

i want to get the first element of each line in this RDD which represent the name of my topic and give it as name of table in saveToCassandra

any ideas please this is my code without what i want

val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
    .map(_._2)
messages.foreachRDD(rdd => {

                    val stockParsed = rdd.map(line => line.split(','))
                    .map(s => new Stock(s(0).toString, s(1).toString, s(2).toString, s(3).toString))

                    //here i want to give the fisrt element as name of table 
                    // if i have Stock(test1,2017/07/23 00:01:02,14,Status)  table = test1
                    // if i have Stock(test2,2017/07/23 00:01:02,14,Status)  table = test2
                    .saveToCassandra("sparkcassandra", table, SomeColumns("topic_name", "date_time", "mesure",                      "status"))
        })       
         ssc.start()
         ssc.awaitTermination()

    }
}  

来源:https://stackoverflow.com/questions/45260052/saving-stream-data-into-tables-cassandra-with-some-names-of-topics

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!