pyspark

问题 I have a csv with 4 columns. The file contains some missing rows based on the series. Input:- No A B C 1 10 50 12 3 40 50 12 4 20 60 15 6 80 80 18 Output:- No A B C 1 10 50 12 2 10 50 12 3 40 50 12 4 20 60 15 5 20 60 15 6 80 80 18 I need pyspark code to generate the above output. 来源： https://stackoverflow.com/questions/60681807/fill-in-missing-values-based-on-series-and-populate-second-row-based-on-previous

问题 I have some json that's being read from a file where each row looks something like this: { "id": "someGuid", "data": { "id": "someGuid", "data": { "players": { "player_1": { "id": "player_1", "locationId": "someGuid", "name": "someName", "assets": { "assetId1": { "isActive": true, "playlists": { "someId1": true, "someOtherId1": false } }, "assetId2": { "isActive": true, "playlists": { "someId1": true } } } }, "player_2": { "id": "player_2", "locationId": "someGuid", "name": "someName", "dict"

问题 I tried taking a schema as a common schema by df.schema() and load all the CSV files to it .But fails as to the assigned schema , the headers of other CSV files doesnot match Any suggestions would be appreciated. as in a function or spark script 回答1: as I understand it. You want to Union / Merge files with different schemas ( though subset of one Master Schema) .. I wrote this function UnionPro which I think just suits your requirement - def unionPro(DFList: List[DataFrame], spark: org.apache

问题 I tried taking a schema as a common schema by df.schema() and load all the CSV files to it .But fails as to the assigned schema , the headers of other CSV files doesnot match Any suggestions would be appreciated. as in a function or spark script 回答1: as I understand it. You want to Union / Merge files with different schemas ( though subset of one Master Schema) .. I wrote this function UnionPro which I think just suits your requirement - def unionPro(DFList: List[DataFrame], spark: org.apache

问题 I had to apply Levenshtein Function on last column when passport and country are same. matrix = passport_heck.select(\ f.col('name_id').alias('name_id_1'), f.col('last').alias('last_1'), f.col('country').alias('country_1'), f.col('passport').alias('passport_1')) \ .crossJoin(passport_heck.select(\ f.col('name_id').alias('name_id_2'), f.col('last').alias('last_2'), f.col('country').alias('country_2'), f.col('passport').alias('passport_2')))\ .filter((f.col('passport_1') == f.col('passport_2'))

问题 Let's say I have a PySpark data frame, like so: +--+--+--+--+ |a |b |c |d | +--+--+--+--+ |1 |0 |1 |2 | |0 |2 |0 |1 | |1 |0 |1 |2 | |0 |4 |3 |1 | +--+--+--+--+ How can I create a column marking all of the duplicate rows, like so: +--+--+--+--+--+ |a |b |c |d |e | +--+--+--+--+--+ |1 |0 |1 |2 |1 | |0 |2 |0 |1 |0 | |1 |0 |1 |2 |1 | |0 |4 |3 |1 |0 | +--+--+--+--+--+ I attempted it using the groupBy and aggregate functions to no avail. 回答1: Just to expand on my comment: You can group by all of

问题 Let's say I have a PySpark data frame, like so: +--+--+--+--+ |a |b |c |d | +--+--+--+--+ |1 |0 |1 |2 | |0 |2 |0 |1 | |1 |0 |1 |2 | |0 |4 |3 |1 | +--+--+--+--+ How can I create a column marking all of the duplicate rows, like so: +--+--+--+--+--+ |a |b |c |d |e | +--+--+--+--+--+ |1 |0 |1 |2 |1 | |0 |2 |0 |1 |0 | |1 |0 |1 |2 |1 | |0 |4 |3 |1 |0 | +--+--+--+--+--+ I attempted it using the groupBy and aggregate functions to no avail. 回答1: Just to expand on my comment: You can group by all of

问题 I have a dataframe like this data = [(("ID1", ['October', 'September', 'August'])), (("ID2", ['August', 'June', 'May'])), (("ID3", ['October', 'June']))] df = spark.createDataFrame(data, ["ID", "MonthList"]) df.show(truncate=False) +---+----------------------------+ |ID |MonthList | +---+----------------------------+ |ID1|[October, September, August]| |ID2|[August, June, May] | |ID3|[October, June] | +---+----------------------------+ I want to compare every row with a default list, such that

问题 I have a spark ec2 cluster where I am submitting a pyspark program from a Zeppelin notebook. I have loaded the hadoop-aws-2.7.3.jar and aws-java-sdk-1.11.179.jar and place them in the /opt/spark/jars directory of the spark instances. I get a java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException Why is spark not seeing the jars? Do I have to have to jars in all the slaves and specify a spark-defaults.conf for the master and slaves? Is there something that needs to be configured

问题 I have a spark ec2 cluster where I am submitting a pyspark program from a Zeppelin notebook. I have loaded the hadoop-aws-2.7.3.jar and aws-java-sdk-1.11.179.jar and place them in the /opt/spark/jars directory of the spark instances. I get a java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException Why is spark not seeing the jars? Do I have to have to jars in all the slaves and specify a spark-defaults.conf for the master and slaves? Is there something that needs to be configured