TBLPROPERTIES('skip.header.line.count'='1') is not working on sparkThrift connected from beeline with hive jdbc 1.2.1

谁说胖子不能爱 提交于 2020-01-06 06:46:21

问题


I am using spark 2.3 and connecting sparkThrift with beeline.

Hive jdbc version 1.2.1 Spark SQL version 2.3.1

I am trying to create external table with skip header property but select command is always returning data with header as first row, below is my create query

CREATE EXTERNAL TABLE datasourcename11(
`retail_invoice_detail_sys_invoice_no` STRING,
`store_id` STRING,
`retail_invoice_detail_invoice_time` STRING,
`retail_invoice_detail_invoice_date` string,
`cust_id` STRING,
`article_code` INTEGER,
`retail_invoice_detail_base_price` INTEGER,
`retail_invoice_detail_sale_price` INTEGER,
`retail_invoice_detail_quantity` DOUBLE,
`retail_invoice_detail_total_amount` DOUBLE
) 
ROW FORMAT DELIMITED  FIELDS TERMINATED BY ',' 
LINES TERMINATED BY '\n'  
LOCATION '/home/java_services/backend/demo/' 
TBLPROPERTIES('skip.header.line.count'=1);

回答1:


This property skip.header.line.count=1 is supported in Hive only.

The workaround is to use filter

retail_invoice_detail_sys_invoice_no!=<col name in header>



来源:https://stackoverflow.com/questions/54540717/tblpropertiesskip-header-line-count-1-is-not-working-on-sparkthrift-connec

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!