Saving twitter streams into a single file with spark streaming, scala

浪子不回头ぞ 提交于 2019-12-11 02:46:01

问题


So after help from this answer Spark Streaming : Join Dstream batches into single output Folder I was able to create a single file for my twitter streams. However,now I don't see any tweets being saved in this file. Please find below my code snippet for this. What am I doing wrong?

        val ssc = new StreamingContext(sparkConf, Seconds(5))
        val stream = TwitterUtils.createStream(ssc, None, filters)
        val tweets = stream.map(r => r.getText)

        tweets.foreachRDD{rdd =>    
        val sqlContext = SQLContextSingleton.getInstance(rdd.sparkContext)
        import sqlContext.implicits._
        val df = rdd.map(t => Record(t)).toDF()
        df.save("com.databricks.spark.csv",SaveMode.Append,Map("path"->"tweetstream.csv")
}
        ssc.start()
        ssc.awaitTermination() 
    }

来源:https://stackoverflow.com/questions/31693971/saving-twitter-streams-into-a-single-file-with-spark-streaming-scala

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!