save Spark dataframe to Hive: table not readable because “parquet not a SequenceFile”

Question

I\'d like to save data in a Spark (v 1.3.0) dataframe to a Hive table using PySpark.

The documentation states:

\"spark.sql.hive.convertMetasto

Answer 1

I have done in pyspark, spark version 2.3.0 :

create empty table where we need to save/overwrite data like:

create table databaseName.NewTableName like databaseName.OldTableName;

then run below command:

df1.write.mode("overwrite").partitionBy("year","month","day").format("parquet").saveAsTable("databaseName.NewTableName");

The issue is you can't read this table with hive but you can read with spark.