apache-spark

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I am using graphframes in pyspark for some graph type of analytics and wondering what would be the best way to create the edge list data frame from a vertices data frame. For example, below is my vertices data frame. I have a list of ids and they belong to different groups. +---+-----+ |id |group| +---+-----+ |a |1 | |b |2 | |c |1 | |d |2 | |e |3 | |a |3 | |f |1 | +---+-----+ My objective is to create an edge list data frame to indicate ids which appear in common groups. Please note that 1 id

问题 I'm trying to transform spark dataframe with 10k rows by latest spark 3.0.1 function mapInPandas. Expected output: mapped pandas_function() transforms one row to three, so output transformed_df should have 30k rows Current output: I'm getting 3 rows with 1 core and 24 rows with 8 cores. INPUT: respond_sdf has 10k rows +-----+-------------------------------------------------------------------+ |url |content | +-----+-------------------------------------------------------------------+ |api_1|{

问题 I have got a requirement to do , but I am confused how to do it. I have two dataframes. so first time i got the below data file1 file1 prodid, lastupdatedate, indicator 00001,,A 00002,01-25-1981,A 00003,01-26-1982,A 00004,12-20-1985,A the output should be 0001,1900-01-01, 2400-01-01, A 0002,1981-01-25, 2400-01-01, A 0003,1982-01-26, 2400-01-01, A 0004,1985-12-20, 2400-10-01, A Second time i got another one file2 prodid, lastupdatedate, indicator 00002,01-25-2018,U 00004,01-25-2018,U 00006,01

问题 I have got a requirement to do , but I am confused how to do it. I have two dataframes. so first time i got the below data file1 file1 prodid, lastupdatedate, indicator 00001,,A 00002,01-25-1981,A 00003,01-26-1982,A 00004,12-20-1985,A the output should be 0001,1900-01-01, 2400-01-01, A 0002,1981-01-25, 2400-01-01, A 0003,1982-01-26, 2400-01-01, A 0004,1985-12-20, 2400-10-01, A Second time i got another one file2 prodid, lastupdatedate, indicator 00002,01-25-2018,U 00004,01-25-2018,U 00006,01

问题 I have the following DataFrame in a Spark (I'm using Scala): [[1003014, 0.95266926], [15, 0.9484202], [754, 0.94236785], [1029530, 0.880922], [3066, 0.7085166], [1066440, 0.69400793], [1045811, 0.663178], [1020059, 0.6274495], [1233982, 0.6112905], [1007801, 0.60937023], [1239278, 0.60044676], [1000088, 0.5789191], [1056268, 0.5747936], [1307569, 0.5676605], [10334513, 0.56592846], [930, 0.5446228], [1170206, 0.52525467], [300, 0.52473146], [2105178, 0.4972785], [1088572, 0.4815367]] I want

问题 I have the following DataFrame in a Spark (I'm using Scala): [[1003014, 0.95266926], [15, 0.9484202], [754, 0.94236785], [1029530, 0.880922], [3066, 0.7085166], [1066440, 0.69400793], [1045811, 0.663178], [1020059, 0.6274495], [1233982, 0.6112905], [1007801, 0.60937023], [1239278, 0.60044676], [1000088, 0.5789191], [1056268, 0.5747936], [1307569, 0.5676605], [10334513, 0.56592846], [930, 0.5446228], [1170206, 0.52525467], [300, 0.52473146], [2105178, 0.4972785], [1088572, 0.4815367]] I want

问题 I have the following DataFrame in a Spark (I'm using Scala): [[1003014, 0.95266926], [15, 0.9484202], [754, 0.94236785], [1029530, 0.880922], [3066, 0.7085166], [1066440, 0.69400793], [1045811, 0.663178], [1020059, 0.6274495], [1233982, 0.6112905], [1007801, 0.60937023], [1239278, 0.60044676], [1000088, 0.5789191], [1056268, 0.5747936], [1307569, 0.5676605], [10334513, 0.56592846], [930, 0.5446228], [1170206, 0.52525467], [300, 0.52473146], [2105178, 0.4972785], [1088572, 0.4815367]] I want

问题 i am new to spark. We are currently building a pipeline : Read the events from Kafka topic Enrich this data with the help of Redis-Lookup Write events to the new Kafka topic So, my problem is when i want to use spark-redis library it performs very well, but data stays static in my streaming job. Although data is refreshed at Redis, it does not reflect to my dataframe. Spark reads data at first then never updates it. Also i am reading from REDIS data at first,total data about 1mio key-val