pyspark

问题 This question already has answers here : Adding a new column in Data Frame derived from other columns (Spark) (3 answers) Closed 2 years ago . I have a data frame in pyspark like sample below. I would like to duplicate a column in the data frame and rename to another column name. Name Age Rate Aira 23 90 Ben 32 98 Cat 27 95 Desired output is : Name Age Rate Rate2 Aira 23 90 90 Ben 32 98 98 Cat 27 95 95 How can I do it? 回答1: Just df.withColumn("Rate2", df["Rate"]) or (in SQL) SELECT *, Rate AS

问题 This question already has answers here : Adding a new column in Data Frame derived from other columns (Spark) (3 answers) Closed 2 years ago . I have a data frame in pyspark like sample below. I would like to duplicate a column in the data frame and rename to another column name. Name Age Rate Aira 23 90 Ben 32 98 Cat 27 95 Desired output is : Name Age Rate Rate2 Aira 23 90 90 Ben 32 98 98 Cat 27 95 95 How can I do it? 回答1: Just df.withColumn("Rate2", df["Rate"]) or (in SQL) SELECT *, Rate AS

问题 Is it possible to use a broadcasted data frame in the UDF of a pyspark SQl application. My Code calls the broadcasted Dataframe inside a pyspark dataframe like below. fact_ent_df_data = sparkSession.sparkContext.broadcast(fact_ent_df.collect()) def generate_lookup_code(col1,col2,col3): fact_ent_df_count=fact_ent_df_data. select(fact_ent_df_br.TheDate.between(col1,col2), fact_ent_df_br.Ent.isin('col3')).count() return fact_ent_df_count sparkSession.udf.register("generate_lookup_code" ,

问题 Is it possible to use a broadcasted data frame in the UDF of a pyspark SQl application. My Code calls the broadcasted Dataframe inside a pyspark dataframe like below. fact_ent_df_data = sparkSession.sparkContext.broadcast(fact_ent_df.collect()) def generate_lookup_code(col1,col2,col3): fact_ent_df_count=fact_ent_df_data. select(fact_ent_df_br.TheDate.between(col1,col2), fact_ent_df_br.Ent.isin('col3')).count() return fact_ent_df_count sparkSession.udf.register("generate_lookup_code" ,

问题 Is it possible to use a broadcasted data frame in the UDF of a pyspark SQl application. My Code calls the broadcasted Dataframe inside a pyspark dataframe like below. fact_ent_df_data = sparkSession.sparkContext.broadcast(fact_ent_df.collect()) def generate_lookup_code(col1,col2,col3): fact_ent_df_count=fact_ent_df_data. select(fact_ent_df_br.TheDate.between(col1,col2), fact_ent_df_br.Ent.isin('col3')).count() return fact_ent_df_count sparkSession.udf.register("generate_lookup_code" ,

问题 I have the below pyspark df which can be recreated by the code df = spark.createDataFrame([(1, "John Doe", "2020-11-30"),(2, "John Doe", "2020-11-27"),(3, "John Doe", "2020-11-29")], ("id", "name", "date")) +---+--------+----------+ | id| name| date| +---+--------+----------+ | 1|John Doe|2020-11-30| | 2|John Doe|2020-11-27| | 3|John Doe|2020-11-29| +---+--------+----------+ I am looking to create a udf to calculate difference between 2 rows of dates (using Lag function) excluding weekends as

问题 My model has output a DenseVector column, and I'd like to find the argmax. This page suggests this function should be available, but I'm not sure what the syntax should be. Is it df.select("mycolumn").argmax() ? 回答1: I could not find the documents for argmax operation in python. but you can do them by converting them to arrays For pyspark 3.0.0 from pyspark.ml.functions import vector_to_array tst_arr = tst_df.withColumn("arr",vector_to_array(F.col('vector_column'))) tst_max=tst_arr.withColumn

问题 I would like to add a new column to my Pyspark dataframe using a Window function, where the rowsBetween are parametrized by values from columns from the dataframe. I tried date_window.rowsBetween(-(F.lit(2) + offset), -offset) , but Spark tells me ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions. which I did not expect in this case. Is there any way to parametrize rowsBetween using values from

问题 I would like to add a new column to my Pyspark dataframe using a Window function, where the rowsBetween are parametrized by values from columns from the dataframe. I tried date_window.rowsBetween(-(F.lit(2) + offset), -offset) , but Spark tells me ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions. which I did not expect in this case. Is there any way to parametrize rowsBetween using values from

问题 I'm using Rest Data Source and I keep running into an issue with the output saying the following: hope_prms = { 'url' : search_url , 'input' : 'new_view' , 'method' : 'GET' , 'readTimeout' : '10000' , 'connectionTimeout' : '2000' , 'partitions' : '10'} sodasDf = spark.read.format('org.apache.dsext.spark.datasource.rest.RestDataSource').options(**hope_prms).load() An error occurred while calling o117.load. : java.lang.ClassNotFoundException: Failed to find data source: org.apache.dsext.spark