pyspark

问题 Is it possible in pyspark to create dictionary within groupBy.agg() ? Here is a toy example: import pyspark from pyspark.sql import Row import pyspark.sql.functions as F sc = pyspark.SparkContext() spark = pyspark.sql.SparkSession(sc) toy_data = spark.createDataFrame([ Row(id=1, key='a', value="123"), Row(id=1, key='b', value="234"), Row(id=1, key='c', value="345"), Row(id=2, key='a', value="12"), Row(id=2, key='x', value="23"), Row(id=2, key='y', value="123")]) toy_data.show() +---+---+-----

问题 I have a udf function: @staticmethod @F.udf("array<int>") def create_users_array(val): """ Takes column of ints, returns column of arrays containing ints. """ return [val for _ in range(val)] I call it like so: df.withColumn("myArray", create_users_array(df["myNumber"])) I pass it a dataframe column of integers, and it returns an array of that integer. E.g. 4 --> [4,4,4,4] It was working until we upgraded from Python 2.7, and upgraded our EMR version (which I believe uses Pyspark 2.3) Anyone

问题 I am using spark 2.4.5 and I need to calculate the sentiment score from a token list column ( MeaningfulWords column) of df1 , according to the words in df2 (spanish sentiment dictionary). In df1 I must create a new column with the scores list of tokens and another column with the mean of scores (sum of scores / count words) of each record. If any token in the list ( df1 ) is not in the dictionary ( df2 ), zero is scored. The Dataframes looks like this: df1.select("ID","MeaningfulWords").show

问题 I am using spark 2.4.5 and I need to calculate the sentiment score from a token list column ( MeaningfulWords column) of df1 , according to the words in df2 (spanish sentiment dictionary). In df1 I must create a new column with the scores list of tokens and another column with the mean of scores (sum of scores / count words) of each record. If any token in the list ( df1 ) is not in the dictionary ( df2 ), zero is scored. The Dataframes looks like this: df1.select("ID","MeaningfulWords").show

问题 I have a bit complex pipeline - pyspark which takes 20 minutes to come up with execution plan. Since I have to execute the same pipeline multiple times with different data frame (as source) Im wondering is there any option for me to avoid building execution plan every time? Build execution plan once and reuse it with different source data?` 回答1: There is a way to do what you ask but it requires advanced understanding of Spark internals. Spark plans are simply trees of objects. These trees are

问题 I'm very new to pyspark/Apache Spark. I need to fetch multiple tables from a database on a server each containing around 120 million rows or more for analysis. I should be able to perform computations on the data. I am running pyspark on a server acting as both master and slave and has 7.45G of RAM. I have installed the jdbc driver and this is the code that I've used. from pyspark.sql import SQLContext from pyspark import SparkContext sc = SparkContext.getOrCreate() sqlContext = SQLContext(sc

问题 I have a pyspark dataframe which contains a column "student" as follows: "student" : { "name" : "kaleem", "rollno" : "12" } Schema for this in dataframe is : structType(List( name: String, rollno: String)) I need to modify this column as "student" : { "student_details" : { "name" : "kaleem", "rollno" : "12" } } Schema for this in dataframe must be : structType(List( student_details: structType(List( name: String, rollno: String)) )) How to do this in spark? 回答1: Use named_struct function to

问题 I'll try to keep it simple. I periodically read some data from a kafka producer and output the following using Spark Structured streaming I have data that outputs like this: +------------------------------------------+-------------------+--------------+-----------------+ |window |timestamp |Online_Emp |Available_Emp | +------------------------------------------+-------------------+--------------+-----------------+ |[2017-12-31 16:01:00, 2017-12-31 16:02:00]|2017-12-31 16:01:27|1 |0 | |[2017

问题 I'm trying to format my timestamp column to include milliseconds without success. How can I format my time to look like this - 2019-01-04 11:09:21.152 ? I have looked at the documentation and following the SimpleDataTimeFormat , which the pyspark docs say are being used by the to_timestamp function. This is my dataframe. +--------------------------+ |updated_date | +--------------------------+ |2019-01-04 11:09:21.152815| +--------------------------+ I use the millisecond format without any

问题 I have a pyspark dataframe with two columns: [Row(zip_code='58542', dma='MIN'), Row(zip_code='58701', dma='MIN'), Row(zip_code='57632', dma='MIN'), Row(zip_code='58734', dma='MIN')] How can I make a key:value pair out of the data inside the columns? e.g.: { "58542":"MIN", "58701:"MIN", etc.. } I would like to avoid using collect for performance reasons. I've tried a few things but can't seem to get just the values . 回答1: As Ankin says, you can use a MapType for this: import pyspark from