spark-dataframe

问题 I have a numpy matrix: arr = np.array([[2,3], [2,8], [2,3],[4,5]]) I need to create a PySpark Dataframe from arr . I can not manually input the values because the length/values of arr will be changing dynamically so I need to convert arr into a dataframe. I tried the following code to no success. df= sqlContext.createDataFrame(arr,["A", "B"]) However, I get the following error. TypeError: Can not infer schema for type: <type 'numpy.ndarray'> 回答1: Hope this helps! import numpy as np #sample

问题 Is there a way to calculate the size in bytes of an Apache spark Data Frame using pyspark? 回答1: why don't you just cache the df and then look in the spark UI under storage and convert the units to bytes df.cache() 来源： https://stackoverflow.com/questions/38180140/how-can-you-calculate-the-size-of-an-apache-spark-data-frame-using-pyspark

问题 I want to do something like this: df .withColumn("newCol", <some formula>) .filter(s"""newCol > ${(math.min(max("newCol").asInstanceOf[Double],10))}""") Exception I'm getting: org.apache.spark.sql.Column cannot be cast to java.lang.Double Can you please suggest me a way to achieve what i want? 回答1: I assume newCol is already present in df , then: import org.apache.spark.sql.expressions.Window import org.apache.spark.sql.functions._ df .withColumn("max_newCol",max($"newCol").over(Window

问题 I have a spark data frame which can have duplicate columns, with different row values, is it possible to coalesce those duplicate columns and get a dataframe without any duplicate columns example : |name |upload| name| upload1| | null| null|alice| 101| | null| null| bob| 231| |alice| 100| null| null| | bob| 23| null| null| should become - |name |upload| upload1| | alice| null| 101| | bob | null| 231| |alice| 100| null| | bob| 23| null| 回答1: val DF1 = Seq( (None, None, Some("alice"), Some(101)

问题 What is the purpose of the getOrCreate method from SparkContext class? I don't understand when we should use this method. If I have 2 spark applications that are run with spark-submit , and in the main method I instantiate the spark context with SparkContext.getOrCreate , both app will have the same context? Or the purpose is simpler, and the only purpose is when I create a spark app, and I don't want to send the spark context as a parameter to a method, and I will get it as a singleton

问题 What is the purpose of the getOrCreate method from SparkContext class? I don't understand when we should use this method. If I have 2 spark applications that are run with spark-submit , and in the main method I instantiate the spark context with SparkContext.getOrCreate , both app will have the same context? Or the purpose is simpler, and the only purpose is when I create a spark app, and I don't want to send the spark context as a parameter to a method, and I will get it as a singleton

问题 What's the right way to insert DF to Hive Internal table in Append Mode. It seems we can directly write the DF to Hive using "saveAsTable" method OR store the DF to temp table then use the query. df.write().mode("append").saveAsTable("tableName") OR df.registerTempTable("temptable") sqlContext.sql("CREATE TABLE IF NOT EXISTS mytable as select * from temptable") Will the second approach append the records or overwrite it? Is there any other way to effectively write the DF to Hive Internal

问题 I am newbie to Apache Spark. My job is read two CSV files, select some specific columns from it, merge it, aggregate it and write the result into a single CSV file. For example, CSV1 name,age,deparment_id CSV2 department_id,deparment_name,location I want to get a third CSV file with name,age,deparment_name I am loading both the CSV into dataframes. And then able to get the third dataframe using several methods join,select,filter,drop present in dataframe I am also able to do the same using

问题 I am trying to calculate percentile of a column in a DataFrame? I cant find any percentile_approx function in Spark aggregation functions. For e.g. in Hive we have percentile_approx and we can use it in the following way hiveContext.sql("select percentile_approx("Open_Rate",0.10) from myTable); But I want to do it using Spark DataFrame for performance reasons. Sample data set |User ID|Open_Rate| ------------------- |A1 |10.3 | |B1 |4.04 | |C1 |21.7 | |D1 |18.6 | I want to find out how many

问题 What is the equivalent in Pyspark for LIKE operator? For example I would like to do: SELECT * FROM table WHERE column LIKE "*somestring*"; looking for something easy like this (but this is not working): df.select('column').where(col('column').like("*s*")).show() 回答1: You can use where and col functions to do the same. where will be used for filtering of data based on a condition (here it is, if a column is like '%string%' ). The col('col_name') is used to represent the condition and like is