pyspark

问题 When i write my JSON file with dataframe.coalesce(1).write.format('json') on pyspark im not able to change the name of file in the partition Im writing my JSON like that: dataframe.coalesce(1).write.format('json').mode('overwrite').save('path') but im not able to change the name of file in the partition I want the path like that: /folder/my_name.json where 'my_name.json' is a json file 回答1: In spark we can't control name of the file written to the directory. First write the data to the HDFS

问题 When i write my JSON file with dataframe.coalesce(1).write.format('json') on pyspark im not able to change the name of file in the partition Im writing my JSON like that: dataframe.coalesce(1).write.format('json').mode('overwrite').save('path') but im not able to change the name of file in the partition I want the path like that: /folder/my_name.json where 'my_name.json' is a json file 回答1: In spark we can't control name of the file written to the directory. First write the data to the HDFS

问题 When i write my JSON file with dataframe.coalesce(1).write.format('json') on pyspark im not able to change the name of file in the partition Im writing my JSON like that: dataframe.coalesce(1).write.format('json').mode('overwrite').save('path') but im not able to change the name of file in the partition I want the path like that: /folder/my_name.json where 'my_name.json' is a json file 回答1: In spark we can't control name of the file written to the directory. First write the data to the HDFS

问题 When i write my JSON file with dataframe.coalesce(1).write.format('json') on pyspark im not able to change the name of file in the partition Im writing my JSON like that: dataframe.coalesce(1).write.format('json').mode('overwrite').save('path') but im not able to change the name of file in the partition I want the path like that: /folder/my_name.json where 'my_name.json' is a json file 回答1: In spark we can't control name of the file written to the directory. First write the data to the HDFS

问题 When i write my JSON file with dataframe.coalesce(1).write.format('json') on pyspark im not able to change the name of file in the partition Im writing my JSON like that: dataframe.coalesce(1).write.format('json').mode('overwrite').save('path') but im not able to change the name of file in the partition I want the path like that: /folder/my_name.json where 'my_name.json' is a json file 回答1: In spark we can't control name of the file written to the directory. First write the data to the HDFS

问题 I am new to Python and stuck with building a hierarchy out of a relational dataset. It would be of immense help if someone has an idea on how to proceed with this. I have a relational data-set with data like _currentnode, childnode_ root, child1 child1, leaf2 child1, child3 child1, leaf4 child3, leaf5 child3, leaf6 so-on. I am looking for some python or pyspark code to build a hierarchy dataframe like below _level1, level2, level3, level4_ root, child1, leaf2, null root, child1, child3, leaf5

问题 I am new to Python and stuck with building a hierarchy out of a relational dataset. It would be of immense help if someone has an idea on how to proceed with this. I have a relational data-set with data like _currentnode, childnode_ root, child1 child1, leaf2 child1, child3 child1, leaf4 child3, leaf5 child3, leaf6 so-on. I am looking for some python or pyspark code to build a hierarchy dataframe like below _level1, level2, level3, level4_ root, child1, leaf2, null root, child1, child3, leaf5

问题 I have a data frame : A, B, C, D, 201701, 2020001 A, B, C, D, 201801, 2020002 A, B, C, D, 201901, 2020003 expected output : col_A, col_B, col_C ,col_D, min_week ,max_week, min_month, max_month A, B, C, D, 201701, 201901, 2020001, 2020003 What I tried in pyspark- from pyspark.sql import Window import pyspark.sql.functions as psf w1 = Window.partitionBy('A','B', 'C', 'D')\ .orderBy('WEEK','MONTH') df_new = df_source\ .withColumn("min_week", psf.first("WEEK").over(w1))\ .withColumn("max_week",

问题 According the documentation is possible to tell Spark to keep track of "out of scope" checkpoints - those that are not needed anymore - and clean them from disk. SparkSession.builder ... .config("spark.cleaner.referenceTracking.cleanCheckpoints", "true") .getOrCreate() Apparently it does so but the problem, however, is that the last checkpointed rdds are never deleted. Question Is there any configuration I am missing to perform all cleanse? If there isn't: Is there any way to get the name of

问题 I have the following dataframe schema below root |-- SOURCE: string (nullable = true) |-- SYSTEM_NAME: string (nullable = true) |-- BUCKET_NAME: string (nullable = true) |-- LOCATION: string (nullable = true) |-- FILE_NAME: string (nullable = true) |-- LAST_MOD_DATE: string (nullable = true) |-- FILE_SIZE: string (nullable = true) I would like to derive a column after extracting the data values from certain columns. The data in location column looks like the following: example 1: prod/docs