spark-dataframe

My DataFrame has the following structure: ------------------------- | Brand | type | amount| ------------------------- | B | a | 10 | | B | b | 20 | | C | c | 30 | ------------------------- I want to reduce the amount of rows by grouping type and amount into one single column of type : Map So Brand will be unique and MAP_type_AMOUNT will have key,value for each type amount combination. I think Spark.sql might have some functions to help in this process, or do I have to get the RDD being the DataFrame and make my "own" conversion to map type? Expected : ------------------------- | Brand | MAP

I was just wondering what people's thoughts were on reading from Hive vs reading from a .csv file or a .txt file or an .ORC file, or a .parquet file. Assuming the underlying Hive table is an external table that has the same file format, would you rather read form a Hive table or from the underlying file itself, and why? Mike tl;dr : I would read it straight from the parquet files I am using Spark 1.5.2 and Hive 1.2.1 For a 5Million row X 100 column table some timings I've recorded are val dffile = sqlContext.read.parquet("/path/to/parquets/*.parquet") val dfhive = sqlContext.table("db.table")

I have a dataframe in Spark 2 as shown below where users have between 50 to thousands of posts. I would like to create a new dataframe that will have all the users in the original dataframe but with only 5 randomly sampled posts for each user. +--------+--------------+--------------------+ | user_id| post_id| text| +--------+--------------+--------------------+ |67778705|44783131591473|some text...........| |67778705|44783134580755|some text...........| |67778705|44783136367108|some text...........| |67778705|44783136970669|some text...........| |67778705|44783138143396|some text...........|

I am trying to calculate percentile of a column in a DataFrame? I cant find any percentile_approx function in Spark aggregation functions. For e.g. in Hive we have percentile_approx and we can use it in the following way hiveContext.sql("select percentile_approx("Open_Rate",0.10) from myTable); But I want to do it using Spark DataFrame for performance reasons. Sample data set |User ID|Open_Rate| ------------------- |A1 |10.3 | |B1 |4.04 | |C1 |21.7 | |D1 |18.6 | I want to find out how many users fall into 10 percentile or 20 percentile and so on. I want to do something like this df.select($"id

I am newbie to Apache Spark. My job is read two CSV files, select some specific columns from it, merge it, aggregate it and write the result into a single CSV file. For example, CSV1 name,age,deparment_id CSV2 department_id,deparment_name,location I want to get a third CSV file with name,age,deparment_name I am loading both the CSV into dataframes. And then able to get the third dataframe using several methods join,select,filter,drop present in dataframe I am also able to do the same using several RDD.map() And I am also able to do the same using executing hiveql using HiveContext I want to

I'm trying to filter one dataframe against another: scala> val df1 = sc.parallelize((1 to 100).map(a=>(s"user $a", a*0.123, a))).toDF("name", "score", "user_id") scala> val df2 = sc.parallelize(List(2,3,4,5,6)).toDF("valid_id") Now I want to filter df1 and get back a dataframe that contains all the rows in df1 where user_id is in df2("valid_id"). In other words, I want all the rows in df1 where the user_id is either 2,3,4,5 or 6 scala> df1.select("user_id").filter($"user_id" in df2("valid_id")) warning: there were 1 deprecation warning(s); re-run with -deprecation for details org.apache.spark

What is the equivalent in Pyspark for LIKE operator? For example I would like to do: SELECT * FROM table WHERE column LIKE "*somestring*"; looking for something easy like this (but this is not working): df.select('column').where(col('column').like("*s*")).show() braj You can use where and col functions to do the same. where will be used for filtering of data based on a condition (here it is, if a column is like '%string%' ). The col('col_name') is used to represent the condition and like is the operator: df.where(col('col1').like("%string%")).show() Using spark 2.0.0 onwards following also