hivecontext

Hive Merge command is not working in Spark HiveContext

随声附和 提交于 2019-12-02 09:28:21
I am running hive merge command using Spark HiveContext in 1.6.3 spark version, but it is failing with below error. 2017-09-11 18:30:33 Driver [INFO ] ParseDriver - Parse Completed 2017-09-11 18:30:34 Driver [INFO ] ParseDriver - Parsing command: MERGE INTO emp_with_orc AS T USING SOURCE_TABLE AS S ON T.id = S.id WHEN MATCHED AND (S.operation = 1) THEN UPDATE SET a = S.a,b = S.b WHEN MATCHED AND (S.operation = 2) THEN DELETE WHEN NOT MATCHED THEN INSERT VALUES (S.id, S.a, S.b) 2017-09-11 18:30:34 Driver [ERROR] HiveWriter - Error while executing the merge query. org.apache.spark.sql

Start HiveThriftServer programmatically in Python

社会主义新天地 提交于 2019-12-02 03:41:33
问题 In the spark-shell (scala), we import, org.apache.spark.sql.hive.thriftserver._ for starting Hive Thrift server programatically for a particular hive context as HiveThriftServer2.startWithContext(hiveContext) to expose a registered temp table for that particular session. How can we do the same using python? Is there a package / api on python for importing HiveThriftServer? Any other thoughts / recommendations appreciated. We have used pyspark for creating a dataframe Thanks Ravi Narayanan 回答1