apache-spark

问题 In a dataframe the columns have the input shown below: | id| priority| status| datetime|data_as_of_Date|Amount|open_close| | 1|Unassigned| Fixed| 10/8/2019 0:00| 2/12/2020 0:00| 40| Closed| | 1|Unassigned| New|2/12/2019 11:00| 2/12/2020 0:00| 20| Open| | 1|Unassigned|Fix in progress|9/12/2019 11:00| 2/12/2020 0:00| 90| Open| | 3| Critical| Removed|5/17/2019 12:00| 2/12/2020 0:00| 33| Closed| | 3|Unassigned|Fix in progress|5/26/2019 10:00| 2/12/2020 0:00| 30| Open| | 3| Critical| New| 5/8/2019

问题 In a dataframe the columns have the input shown below: | id| priority| status| datetime|data_as_of_Date|Amount|open_close| | 1|Unassigned| Fixed| 10/8/2019 0:00| 2/12/2020 0:00| 40| Closed| | 1|Unassigned| New|2/12/2019 11:00| 2/12/2020 0:00| 20| Open| | 1|Unassigned|Fix in progress|9/12/2019 11:00| 2/12/2020 0:00| 90| Open| | 3| Critical| Removed|5/17/2019 12:00| 2/12/2020 0:00| 33| Closed| | 3|Unassigned|Fix in progress|5/26/2019 10:00| 2/12/2020 0:00| 30| Open| | 3| Critical| New| 5/8/2019

问题 I have a datafame and would like to add columns to it, based on values from a list. The list of my values will vary from 3-50 values. I'm new to pySpark and I'm trying to append these values as new columns (empty) to my df. I've seen recommended code of how to add [one column][1] to a dataframe but not multiple from a list. mylist = ['ConformedLeaseRecoveryTypeId', 'ConformedLeaseStatusId', 'ConformedLeaseTypeId', 'ConformedLeaseRecoveryTypeName', 'ConformedLeaseStatusName',

问题 I have a Spark DataFrame , let's say 'df'. I do the following simple aggregation on this DataFrame : df.groupBy().sum() Upon doing so, I get the following exception: java.lang.IllegalArgumentException: requirement failed: Decimal precision 39 exceeds max precision 38 Is there any way I can fix this? My guess is, if I can decrease the decimal precision of all the columns of double type in df, it would solve the problem. 来源： https://stackoverflow.com/questions/46462377/change-decimal-precision

问题 I have a Spark DataFrame , let's say 'df'. I do the following simple aggregation on this DataFrame : df.groupBy().sum() Upon doing so, I get the following exception: java.lang.IllegalArgumentException: requirement failed: Decimal precision 39 exceeds max precision 38 Is there any way I can fix this? My guess is, if I can decrease the decimal precision of all the columns of double type in df, it would solve the problem. 来源： https://stackoverflow.com/questions/46462377/change-decimal-precision

问题 I'm currently dealing with the following source data in a JSON file: { "unique_key_1": { "some_value_1": 1, "some_value_2": 2 }, "unique_key_2": { "some_value_1": 2, "some_value_2": 3 } "unique_key_3": { "some_value_1": 2, "some_value_2": 1 } ... } Note that the source data is effective a large dictionary, with lots of unique keys. It is NOT a list of dictionaries. I have lots of large JSON files like this that I want to parse into the following DataFrame structure using PySpark: key | some

问题 I'm currently dealing with the following source data in a JSON file: { "unique_key_1": { "some_value_1": 1, "some_value_2": 2 }, "unique_key_2": { "some_value_1": 2, "some_value_2": 3 } "unique_key_3": { "some_value_1": 2, "some_value_2": 1 } ... } Note that the source data is effective a large dictionary, with lots of unique keys. It is NOT a list of dictionaries. I have lots of large JSON files like this that I want to parse into the following DataFrame structure using PySpark: key | some

问题 I'm using Spark for fun and to learn new things about MapReduce. So, I'm trying to write a program suggesting new friendships (i.e., a sort of recommendation system). The suggestion of a friendship between two individuals is performed if they are not connected yet and have a lot of friends in common. The friendship text file has a structure similar to the following: 1 2,4,11,12,15 2 1,3,4,5,9,10 3 2,5,11,15,20,21 4 1,2,3 5 2,3,4,15,16 ... where the syntax is: ID_SRC1<TAB>ID_DST1,ID_DST2,... .

问题 I have a PySpark Dataframe that contains an ArrayType(StringType()) column. This column contains duplicate strings inside the array which I need to remove. For example, one row entry could look like [milk, bread, milk, toast] . Let's say my dataframe is named df and my column is named arraycol . I need something like: df = df.withColumn("arraycol_without_dupes", F.remove_dupes_from_array("arraycol")) My intution was that there exists a simple solution to this, but after browsing stackoverflow

问题 I have a Spark (2.4.0) data frame with a column that has just two values (either 0 or 1 ). I need to calculate the streak of consecutive 0 s and 1 s in this data, resetting the streak to zero if the value changes. An example: from pyspark.sql import (SparkSession, Window) from pyspark.sql.functions import (to_date, row_number, lead, col) spark = SparkSession.builder.appName('test').getOrCreate() # Create dataframe df = spark.createDataFrame([ ('2018-01-01', 'John', 0, 0), ('2018-01-01', 'Paul