pyspark-sql

If I call map or mapPartition and my function receives rows from PySpark what is the natural way to create either a local PySpark or Pandas DataFrame? Something that combines the rows and retains the schema? Currently I do something like: def combine(partition): rows = [x for x in partition] dfpart = pd.DataFrame(rows,columns=rows[0].keys()) pandafunc(dfpart) mydf.mapPartition(combine) zero323 Spark >= 2.3.0 Since Spark 2.3.0 it is possible to use Pandas Series or DataFrame by partition or group. See for example: Applying UDFs on GroupedData in PySpark (with functioning python example)

I have requirement where i need to count number of duplicate rows in SparkSQL for Hive tables. from pyspark import SparkContext, SparkConf from pyspark.sql import HiveContext from pyspark.sql.types import * from pyspark.sql import Row app_name="test" conf = SparkConf().setAppName(app_name) sc = SparkContext(conf=conf) sqlContext = HiveContext(sc) df = sqlContext.sql("select * from DV_BDFRAWZPH_NOGBD_R000_SG.employee") As of now i have hardcoded the table name, but it actually comes as parameter. That being said we don't know the number of columns or their names as well.In python pandas we have

Having some trouble getting the round function in pyspark to work - I have the below block of code, where I'm trying to round the new_bid column to 2 decimal places, and rename the column as bid afterwards - I'm importing pyspark.sql.functions AS func for reference, and using the round function contained within it: output = output.select(col("ad").alias("ad_id"), col("part").alias("part_id"), func.round(col("new_bid"), 2).alias("bid")) the new_bid column here is of type float - the resulting dataframe does not have the newly named bid column rounded to 2 decimal places as I am trying to do,

This question already has an answer here: Spark Equivalent of IF Then ELSE 4 answers I am trying to use a "chained when" function. In other words, I'd like to get more than two outputs. I tried using the same logic of the concatenate IF function in Excel: df.withColumn("device_id", when(col("device")=="desktop",1)).otherwise(when(col("device")=="mobile",2)).otherwise(null)) But that doesn't work since I can't put a tuple into the "otherwise" function. Grr Have you tried: from pyspark.sql import functions as F df.withColumn('device_id', F.when(col('device')=='desktop', 1).when(col('device')==

Is it possible to cache a data frame and then reference (query) it in another script?...My goal is as follows: In script 1, create a data frame (df) Run script 1 and cache df In script 2, query data in df zero323 Spark >= 2.1.0 Since Spark 2.1 you can create global temporary views ( createGlobalTempView ), which can be accessed across multiple sessions using the same metastore, as long as the original session is kept alive: The lifetime of this temporary view is tied to this Spark application. Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e.

I'm using Spark 2.0 with PySpark. I am redefining SparkSession parameters through a GetOrCreate method that was introduced in 2.0: This method first checks whether there is a valid global default SparkSession, and if yes, return that one. If no valid global default SparkSession exists, the method creates a new SparkSession and assigns the newly created SparkSession as the global default. In case an existing SparkSession is returned, the config options specified in this builder will be applied to the existing SparkSession. https://spark.apache.org/docs/2.0.1/api/python/pyspark.sql.html#pyspark

--archives , --files , --py-files and sc.addFile and sc.addPyFile are quite confusing, can someone explain these clearly? These options are truly scattered all over the place. In general, add your data files via --files or --archives and code files via --py-files . The latter will be added to the classpath (c.f., here ) so you could import and use. As you can imagine, the CLI arguments is actually dealt with by addFile and addPyFiles functions (c.f., here ) From http://spark.apache.org/docs/latest/programming-guide.html Behind the scenes, pyspark invokes the more general spark-submit script.

In pandas, this can be done by column.name. But how to do the same when its column of spark dataframe? e.g. The calling program has a spark dataframe: spark_df >>> spark_df.columns ['admit', 'gre', 'gpa', 'rank'] This program calls my function: my_function(spark_df['rank']) In my_function, I need the name of the column i.e. 'rank' If it was pandas dataframe, we can use inside my_function >>> pandas_df['rank'].name 'rank' You can get the names from the schema by doing spark_df.schema.names Printing the schema can be useful to visualize it as well spark_df.printSchema() The only way is to go an

Let's say I have a spark data frame df1, with several columns (among which the column 'id') and data frame df2 with two columns, 'id' and 'other'. Is there a way to replicate the following command sqlContext.sql("SELECT df1.*, df2.other FROM df1 JOIN df2 ON df1.id = df2.id") by using only pyspark functions such as join(), select() and the like? I have to implement this join in a function and I don't want to be forced to have sqlContext as a function parameter. Thanks! Pablo Estevez Not sure if the most efficient way, but this worked for me: from pyspark.sql.functions import col df1.alias('a')