hiveql

I have the use case below : My application has a table having multiyear data in RDBMS DB. We have used sqoop to get data into HDFS and have loaded into hive table partitioned by year, month . Now, the application updates, and inserts new records into RDBMS Table table daily as well. These updated records can span across history months. Updated records and new insert records can be determined by updated timestamp field (it will have current day timestamp). Now the problem here is : how to do delta/incremental load hive table daily using these updated records. -> I know there is a sqoop

I have a hive table, name age sal A 45 1222 B 50 4555 c 44 8888 D 78 1222 E 12 7888 F 23 4555 I want to calculate median of age column. Below is my approach select min(age) as HMIN,max(age) as HMAX,count(age) as HCount, IF(count(age)%2=0,'even','Odd') as PCOUNT from v_act_subjects_bh; Appreciate any query suggestion Amar You can use the percentile function to compute the median. Try this: select percentile(cast(age as BIGINT), 0.5) from table_name 来源： https://stackoverflow.com/questions/26863139/how-to-calculate-median-in-hive

How do we get the current system date in Hive? In MySQL we have select now(), can any one please help me to get the query results. I am very new to Hive, is there a proper documentation for Hive that gives the details information about the pseudo columns, and built-in functions. According to the LanguageManual , you can use unix_timestamp() to get the "current time stamp using the default time zone." If you need to convert that to something more human-readable, you can use from_unixtime(unix_timestamp()) . Hope that helps. Aniket Asati Yes... I am using Hue 3.7.0 - The Hadoop UI and to get

I am working with Spark and PySpark. I am trying to achieve the result equivalent to the following pseudocode: df = df.withColumn('new_column', IF fruit1 == fruit2 THEN 1, ELSE 0. IF fruit1 IS NULL OR fruit2 IS NULL 3.) I am trying to do this in PySpark but I'm not sure about the syntax. Any pointers? I looked into expr() but couldn't get it to work. Note that df is a pyspark.sql.dataframe.DataFrame . There are a few efficient ways to implement this. Let's start with required imports: from pyspark.sql.functions import col, expr, when You can use Hive IF function inside expr: new_column_1 =

I am facing one issue in converting the date in hive. I need to convert 2017-sep-12 To 2017-09-12 . How can i achieve this in HIVE Use unix_timestamp(string date, string pattern) to convert given date format to seconds passed from 1970-01-01. Then use from_unixtime() to convert to given format: hive> select from_unixtime(unix_timestamp('2017-sep-12' ,'yyyy-MMM-dd'), 'dd-MM-yyyy'); OK 12-09-2017 来源： https://stackoverflow.com/questions/47301455/how-to-convert-date-2017-sep-12-to-2017-09-12-in-hive

My question is : Is there a way to do find the last day of a month in Hive, like Oracle SQL function ? : LAST_DAY(D_Dernier_Jour) Thanks. You could make use of last_day(dateString) UDF provided by Nexr. It returns the last day of the month based on a date string with yyyy-MM-dd HH:mm:ss pattern. Example: SELECT last_day('2003-03-15 01:22:33') FROM src LIMIT 1; 2003-03-31 00:00:00 You need to pull it from their Github Repository and build. Their wiki page contains all the info on how to build and use it with Hive. HTH As of Hive 1.1.0, last_day(string date) function is available. last_day

I would like to know if the order of the data records matter (performance wise) when joining two tables? P.S. I am not using any map-side join or bucket join. Thank you! On the one hand order should not matter because during shuffle join files are being read by mappers in parallel, also files may be splitted between few mappers or vice-versa, one mapper can read few files, then mappers output passed to each reducer. And even if data was ordered it is being read and distributed not in it's order due to parallelism . On the other hand, ordering data may improve compression depending on the data