SparkSQL error Table Not Found

Question

I converted an RDD[myClass] to dataframe and then register it as an SQL table

my_rdd.toDF().registerTempTable(\"my_rdd\")

Answer 1

I found it easy to cause problems with temptables if there is more than one open zeppelin session, either in your own browser, or from someone else using the same server. The variable sqlContext is shared across those sessions and its easy to overwrite the value of it.

Answer 2

I faced with a similar problem. I was loading a table which was not present in the warehouse folder whereas Hive console was showing me the table name. You can check the detailed description of the table you are loading using describe formatted table_name. You don't need to copy any file to spark/conf folder. It is already integrated.

Answer 3

I have met the same error but in different case ,by solve it with using the same context. If you use hiveContext, make sure that you use it all the time, for example first sqlContext.sql("load data input XXX") , and then if you use hiveContext.sql("select * from XXX"), you will meet this problem.

Every context has it`s lifecycle. So do not use two context with the same dataFrame .

Answer 4

So as the solution of this problem I copied core-site.xml,hive-site.xml and hdfs-site.xml files into conf directoy.

Answer 5

Make sure to import the implicits._ from the same SQLContext. Temporary tables are kept in-memory in one specific SQLContext.

val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
my_rdd.toDF().registerTempTable("my_rdd")

val my_df = sqlContext.sql("SELECT * from my_rdd LIMIT 5")
my_df.collect().foreach(println)