pyspark

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I'm trying to transform spark dataframe with 10k rows by latest spark 3.0.1 function mapInPandas. Expected output: mapped pandas_function() transforms one row to three, so output transformed_df should have 30k rows Current output: I'm getting 3 rows with 1 core and 24 rows with 8 cores. INPUT: respond_sdf has 10k rows +-----+-------------------------------------------------------------------+ |url |content | +-----+-------------------------------------------------------------------+ |api_1|{

问题 I have spark application which read multiple s3 files and do certain transformation. This is how I am reading the files: input_df_s3_path = spark.read.csv("s3a://bucket1/s3_path.csv") s3_path_list = input_df_s3_path.select('_c0').rdd.map(lambda row : row[0]).collect() input_df = sqlContext.read.option("mergeSchema", "false").parquet(*s3_path_list).na.drop() So creating a datafrme from a csv which consists all the s3 path, converting those paths into a list and passing that list in read

问题 I have a pyspark Dataframe , I would like to join 3 columns. id | column_1 | column_2 | column_3 -------------------------------------------- 1 | 12 | 34 | 67 -------------------------------------------- 2 | 45 | 78 | 90 -------------------------------------------- 3 | 23 | 93 | 56 -------------------------------------------- I want to join the 3 columns : column_1, column_2, column_3 in only one adding between there value "-" Expect result: id | column_1 | column_2 | column_3 | column_join -

问题 I have a pyspark Dataframe , I would like to join 3 columns. id | column_1 | column_2 | column_3 -------------------------------------------- 1 | 12 | 34 | 67 -------------------------------------------- 2 | 45 | 78 | 90 -------------------------------------------- 3 | 23 | 93 | 56 -------------------------------------------- I want to join the 3 columns : column_1, column_2, column_3 in only one adding between there value "-" Expect result: id | column_1 | column_2 | column_3 | column_join -

问题 Hi I'm making transformation, I have created some_function(iter) generator to yield Row(id=index, api=row['api'], A=row['A'], B=row['B'] to yield transformed rows from pandas dataframe to rdd and to spark dataframe. I'm getting errors. (I must use pandas to transform data as there is a large amount of legacy code) Input Spark DataFrame respond_sdf.show() +-------------------------------------------------------------------+ |content | +----------------------------------------------------------

问题 I would like to use a tensorflow.keras model in a pysark pandas_udf. However, I get a pickle error when the model is being serialized before sending it to the workers. I am not sure I am using the best method to perform what I want, therefore I will expose a minimal but complete example. Packages: tensorflow-2.2.0 (but error is triggered to all previous versions too) pyspark-2.4.5 The import statements are: import pandas as pd import numpy as np from tensorflow.keras.models import Sequential

问题 So I have a dataframe and I want to calculation some quantity let's say in daily basis..let's say we have 10 columns col1,col2,col3,col4... coln which each columns are dependent on value col1 , col2, col3 , col4.. and so on and the date resets based on the id .. +--------+----+---- +----+ date |col1|id |col2|. . |coln +--------+----+---- +----+ 2020-08-01| 0| M1 | . . . 3| 2020-08-02| 4| M1 | 10| 2020-08-03| 3| M1 | . . . 9 | 2020-08-04| 2| M1 | . . . 8 | 2020-08-05| 1| M1 | . . . 7 | 2020-08