How to connect Spark SQL to remote Hive metastore (via thrift protocol) with no hive-site.xml?

后端 未结 8 1362
我寻月下人不归
我寻月下人不归 2020-11-22 12:07

I\'m using HiveContext with SparkSQL and I\'m trying to connect to a remote Hive metastore, the only way to set the hive metastore is through including the hive-site.xml on

8条回答
  •  余生分开走
    2020-11-22 12:49

    Spark Version : 2.0.2

    Hive Version : 1.2.1

    Below Java code worked for me to connect to Hive metastore from Spark:

    import org.apache.spark.sql.SparkSession;
    
    public class SparkHiveTest {
    
        public static void main(String[] args) {
    
            SparkSession spark = SparkSession
                      .builder()
                      .appName("Java Spark Hive Example")
                      .config("spark.master", "local")
                      .config("hive.metastore.uris",                
                       "thrift://abc123.com:9083")
                      .config("spark.sql.warehouse.dir", "/apps/hive/warehouse")
                      .enableHiveSupport()
                      .getOrCreate();
    
            spark.sql("SELECT * FROM default.survey_data limit 5").show();
        }
    }
    

提交回复
热议问题