SQLContext.gerorCreate is not a value

左心房为你撑大大i 提交于 2019-12-25 17:58:06

问题


I am getting error SQLContext.gerorCreate is not a value of object org.apache.spark.SQLContext. This is my code

import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.sql.functions
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.types
import org.apache.spark.SparkContext
import java.io.Serializable
case class Sensor(id:String,date:String,temp:String,press:String)
object consum {
 def main(args: Array[String]) {
  val sparkConf = new SparkConf().setAppName("KafkaWordCount").setMaster("local[2]")
val ssc = new StreamingContext(sparkConf, Seconds(2))
val sc=new SparkContext(sparkConf) 
val lines = KafkaUtils.createStream(ssc, "localhost:2181", "spark-streaming-consumer-group", Map("hello" -> 5))


def parseSensor(str:String): Sensor={
    val p=str.split(",")
    Sensor(p(0),p(1),p(2),p(3))
  }
val data=lines.map(_._2).map(parseSensor)
val sqlcontext=new SQLContext(sc)

import sqlcontext.implicits._
data.foreachRDD { rdd=>



val sensedata=sqlcontext.getOrCreate(rdd.sparkContext) 
}

I have tried with SQLContext.getOrCreate as well but same error.


回答1:


There is no such getOrCreate function defined for neither SparkContext nor SQLContext.

getOrCreate function is defined for SparkSession instances from which SparkSession instances are created. And we get sparkContext instance or sqlContext instance from the SparkSession instance created using getOrCreate method call.

I hope the explanation is clear.

Updated

The explanation I did above is suitable for higher versions of spark. In the blog as the OP is referencing, the author is using spark 1.6 and the api doc of 1.6.3 clearly states

Get the singleton SQLContext if it exists or create a new one using the given SparkContext



来源:https://stackoverflow.com/questions/48968847/sqlcontext-gerorcreate-is-not-a-value

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!